Bug#1036731: python3-debian: fails to parse some debian/control.in files

[Jelmer Vernooij]

On Wed, May 24, 2023 at 10:56:21PM +0200, Dylan Aïssi wrote:
> python3-debian is not able anymore to parse correctly some debian/control.in.
> The first version with this issue is 0.1.44, so I suppose it is related to the
> new RTS parser. The consequence of this bug is wrap-and-sort crashes when it
> run on these files. Below is the error message:
> 
> Traceback (most recent call last):
>   File "/usr/bin/wrap-and-sort", line 496, in <module>
>     main()
>   File "/usr/bin/wrap-and-sort", line 481, in main
>     modified_files = wrap_and_sort(args)
>                      ^^^^^^^^^^^^^^^^^^^
>   File "/usr/bin/wrap-and-sort", line 312, in wrap_and_sort
>     control = WrapAndSortControl(control_file, args)
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/bin/wrap-and-sort", line 99, in __init__
>     super().__init__(filename, use_rts_parser=args.rts_parser)
>   File "/usr/lib/python3/dist-packages/devscripts/control.py", line
> 210, in __init__
>     self._deb822_file = parse_deb822_file(sequence)
>                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/parsing.py",
> line 3095, in parse_deb822_file
>     deb822_file = Deb822FileElement(LinkedList(tokens))
>                                     ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_util.py", line 159, in __init__
>     self.extend(values)
>   File "/usr/lib/python3/dist-packages/debian/_util.py", line 272, in extend
>     for v in values:
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 104, in _impl
>     for token in token_stream:
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 104, in _impl
>     for token in token_stream:
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/parsing.py",
> line 2991, in _build_field_with_value
>     value_element = next(buffered_stream, None)
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 143, in __next__
>     return next(self._stream)
>            ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 104, in _impl
>     for token in token_stream:
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/parsing.py",
> line 2939, in _build_value_line
>     tokens_in_value = list(buffered_stream.takewhile(_non_end_of_line_token))
>                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 149, in takewhile
>     while buffer or self._fill_buffer(5):
>                     ^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 198, in _fill_buffer
>     self._buffer.append(next(self._stream))
>                         ^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/_util.py",
> line 104, in _impl
>     for token in token_stream:
>   File "/usr/lib/python3/dist-packages/debian/_deb822_repro/parsing.py",
> line 3031, in _abort_on_error_tokens
>     raise SyntaxOrParseError(
> debian._deb822_repro.types.SyntaxOrParseError: Syntax or Parse error
> on the line: "%if USE_SYSTEM_ZLIB\n"
> 
> An easy way to reproduce the crash is to run wrap-and-sort on the
> debian/ folder of
> firefox-esr:
> > wget http://deb.debian.org/debian/pool/main/f/firefox-esr/firefox-esr_102.11.0esr-1.debian.tar.xz
> > tar -xvf firefox-esr
> > wrap-and-sort
> 
> It crashes in a similar way on these packages: babel-minify and xapian-bindings.

Thanks for the bug report.

FWIW Deb822 isn't really built to support parsing deb822 files with
arbitrary data in the middle. This means it doesn't do well with at
least some .in files (which use a variety of styles).

That said, it would be good to handle this particular case better.

Cheers,

Jelmer