Bug#1068174: yosys: Please package the latest upstream release

Sun Mar 16 16:34:10 GMT 2025

Hi Steffen, Scott, 

On Sat, Mar 15, 2025 at 05:49:30PM +0000, Steffen Möller wrote:
> if the documentation needs another package that is not yet in Debian then
> I suggest to have Yosys uploaded without the documentation.

Nah. I'm too much of a perfectionist for that. I'd sooner switch to HTML
output than have no docs in Debian at all (maybe we should enable those
anyway?).

You probably just missed Scott had this working before so we're not at that
level of desperation quite yet :-).

On Sat, Mar 15, 2025 at 04:30:35PM +0000, Scott Ashcroft wrote:
> On Fri, 2025-03-14 at 21:10 +0100, Daniel Gröber wrote:
> > tl;dr: yosys packaging still has issues in the docs build. Python/sphinx
> > help wanted :-).
> 
> When I try to build the package I get:
> 
> $ gbp buildpackage
> gbp:info: Creating /home/scott/yosys_0.49.orig.tar.gz
> gbp:error: Error creating yosys_0.49.orig.tar.gz: Pristine-tar couldn't
> checkout "yosys_0.49.orig.tar.gz": fatal: ambiguous argument
> '20a7873377c94821e2eeda737aaf35fab389d8cc^{tree}': unknown revision or

I think I just rebased after generating the pristine-tar commit like an
idiot. It works on my system because my repo still has the tree objects but
salsa doesn't because they're not referenced by any head (anymore) and so
didn't get pushed. Force-pushed a corrected pristine-tar branch, should
work now.

This is (partly) a design flaw in pristine-tar. It should be generating
merge commits for the metadata branch so git knows to keep the trees it
uses instead of hardcoding dangling tree IDs in a file. Gah. We have a
saying in German (Steffen's going to appreciate this) that applies just as
well to pristine-tar's design as it does to me in this case:

  Einmal mit profis...

I was just about to switch yosys to pristine-lfs but gbp doesn't support
that natively yet so it would be more confusing than useful.

WIP PR from Andrew to add LSF support this is here:
https://github.com/agx/git-buildpackage/pull/79

I'm prety done with these workflow problems. The only viable solution I see
is dpkg source format 3.0 (git). I hope I can get around to doing some work
to finish the implementation and make it agreeable to ftp-master. That's a
TODO for DebConf I suppose.

If you're not familliar see dpkg-source(1) Section "Format: 3.0 (git)". It
essentially replaces all the awful tarball hacks necessary when working
with git upstreams by dsc source packages natively made of git-bundle(1)s.

Recent activity on tag2upload makes me hopeful that could politically
happen yet since it would also nicely complement that movement by removing
the impedenace mismatch between git and format 3.0 (quilt) dscs which is
currently solved by package content restrictions and awfully nasty
conversion hacks.

> > Now when I build in sbuild I still get a number of failures. For one
> > Makefile's rsync dependency isn't declared. Easy fix already done. This
> > type of thing is why we usually build with some kind of isolated builder
> > and not using debuild/dpkg-buildpackage directly on our workstation.
> 
> I'm setting up sbuild now. For some reason I thought that using gbp
> sorted out the isolation but that's obviously not the case.

Just in case you haven't found it: sbuild-debian-developer-setup is quite
convenient for a fire-and-forget setup while still keeping chroots updated
using the included cron job.

> > Then build-indep breaks because of python problems:
> > 
> > ```
> > ./yosys-smtbmc --help > docs/source/generated/yosys-smtbmc
> > Traceback (most recent call last):
> >   File "/build/yosys-ghivvZ/yosys-0.49/./yosys-smtbmc", line 22, in <module>
> >     from smtio import SmtIo, SmtOpts, MkVcd
> > ModuleNotFoundError: No module named 'smtio'
> > make: *** [Makefile:1034: docs/source/generated/yosys-smtbmc] Error 1
> > dh_auto_build: error: make -j1 "INSTALL=install --strip-program=true" docs DOC_TARGET=latexpdf -j1 returned exit code 2
> 
> 
> I didn't see that as my non-isolated builds would have picked up the
> old versions of the python files from /usr/share/yosys since I had
> yosys installed.

Ah, that makes sense :D.

> I think the fix will be to set $PYTHONPATH to point to the correct
> directory in the source while the docs are being built. That way the
> installed code won't have random paths embedded in it.

Sounds reasonable. I just wonder how upstream deals with these issues, or
put another way, why we're seeing this and they don't.

I never felt like digging into the python patches because of the horrors
I've seen in that ecosystem ;P.

On Sat, Mar 15, 2025 at 08:38:59PM +0000, Scott Ashcroft wrote:
> If you cherry pick the four following commits from:
> 
> https://salsa.debian.org/sashcroft/yosys
> 
> 5f1c9942c4d26295f620535b8c9dd7da9df4aeb3
> 142b85a3c02af3b9646abdda8ff5b54d4d44385f
> b688f952624ac5ca74e3a0c46c85797d91873ddd
> 306735c164c350a85bb2cb44168582e316974ee7

Yup, those seems to have done it.

    Build Architecture: amd64
    Job: /home/dxld/share/dev/deb/pkg/yosys_0.49-1.dsc
    Lintian: ***pass***
    Status: ***successful***

Now on to hard mode: autpkgtest. It seems debian/yosys-testsuite has lost
it's precision. It runs through sucessfully but doesn't actually do
anything. Upstream seems to have made some high-concept makefile changes
that are breaking:

    # makefile-tests/ is a dummy string, not a directory
    .PHONY: makefile-tests
    makefile-tests: $(MK_TEST_DIRS:%=makefile-tests/%)
    # this target actually emits .mk files
    %.mk:
        +cd $(dir $*) && bash run-test.sh
    # this one spawns submake on each
    makefile-tests/%: %/run-test.mk $(TARGETS) $(EXTRA_TARGETS)
        $(MAKE) -C $* -f run-test.mk
        + at echo "...passed tests in $*"

    test: makefile-tests abcopt-tests seed-tests

Haven't got the spoons remaining to figure that out today. Given a working
sbuild setup autopkgtest is easy (if obscure) to run if you want to give
this a shot. To re-use the binary .debs you already have:

    autopkgtest -B -s . -- schroot chroot:unstable-amd64-sbuild

-s is to get a shell on test failure so fail it intentially if you want to
explore in the test environment.

Also I have a note on debian/changelog. You should always use "UNRELEASED"
in the distribution field as it has special handling in tools.

For example dch will create another release when the current entry isn't
UNRELEASED (instead of editing the current one) and lintian skips some
"invalid distribution" checks which otherwise show up as noise to filter
though.

The trick to get that to play nice with sbuild's chroot detection is having
appropriate aliases in the schroot config file (which
sbuild-debian-developer-setup does right IIRC):

    aliases=UNRELEASED,UNRELEASED-amd64-sbuild,sid

Usually, with the gbp sponsorship workflow, a DD will then change
UNRELEASED to "unstable" right before gbp tag and uploading.

> That should get everything working. (I'll do a merge request if you
> want but only if that's easier for you)

Nah. This is fine.

> Running the build properly in sbuild made the errors reasonably easy to
> fix. Thanks for your help with all of this.

No, thank *you* for your help <3. 

> The python stuff was just the include path. By fixing it for post-
> install use we'd broken it for use during the build. Now it runs
> correctly I had to make sure dh_missing didn't moan about the python
> cache files which are created but not installed.
> 
> The sphinx warnings were mostly due to use of rsync in another
> Makefile. I've patched it to use cp in a similar way to your patch.
> 
> Getting that working revealed the docs build was trying to download
> some graphics from github. I've patched those calls out.

Upstreams these days... :-)

> I've also partially reverted the || rm $@@ patch as some of the
> commands intentionally error out when called with --help.

Right. Krystal also pointed this out upstream. I still have to replace it
with FORCE (always rerun that target) in our patch.

Thanks,
--Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/debian-science-maintainers/attachments/20250316/f24391d2/attachment.sig>