[Debian-med-packaging] Question about htslib usage in blat / libucsc source

[John Marshall]

On 22 Jan 2020, at 16:26, Andreas Tille <andreas at an3as.eu> wrote:
> I'm currently trying to package blat which is very tricky since it
> contains non-free parts, a private copy of htslib (eliminating this is
> my current intention) and upstream who insists that using a code copy of
> an old version is the right way to go for them.

Obviously in general the HTSlib maintainers also prefer it when people do not bundle ancient versions of HTSlib.

The kent tree's bundled htslib-1.3 contains a number of local patches to vanilla htslib-1.3:

* Add new cram_get_Md5/cram_get_ref_url/cram_get_cache_dir/cram_set_cache_url functions which change HTSlib to use the kent tree's reference path and cache mechanisms instead of HTS_REF and HTS_CACHE

* Switch out knetfile to use the kent tree's networking facilities

* Stop test_and_fetch() from writing a local copy of downloaded indexes

* Various other trivia, much of which has already been applied to more recent vanilla HTSlib (or equivalents have been)

Most of this you can ignore. The only item that causes you much difficulty is the first one, but as a first pass at it you can ignore it and require that users set up both kent's and htslib's variables pointing at the reference sources and cache. This is the cause of the following errors:

> bamFile.c: In function ‘bamFetchAlreadyOpen’:
> bamFile.c:190:24: warning: implicit declaration of function ‘cram_get_Md5’; did you mean ‘cram_get_refs’? [-Wimplicit-function-declaration]
> bamFile.c:197:25: warning: implicit declaration of function ‘cram_get_ref_url’; did you mean ‘cram_get_refs’? [-Wimplicit-function-declaration]
> bamFile.c:198:26: warning: implicit declaration of function ‘cram_get_cache_dir’; did you mean ‘cram_set_header’? [-Wimplicit-function-declaration]
> bamFile.c: In function ‘bamAndIndexFetchPlus’:
> bamFile.c:240:5: warning: implicit declaration of function ‘cram_set_cache_url’ [-Wimplicit-function-declaration]

...as they invoke these functions that exist only in the locally patched bundled htslib. These calls just set up the hacked up bundled htslib to use the kent reference path/cache mechanisms. As you're ignoring that, you should just nuke this code from bamFile.c, as in the attached patch.

> bamFile.c: In function ‘bamGetTargetCount’:
> bamFile.c:264:21: error: ‘SAM_hdr’ {aka ‘struct sam_hdr_t’} has no member named ‘nref’
>  264 |     tCount = cramHdr->nref;

This one is an actual case of using private facilities. It can easily be patched to use proper API functions instead (as attached), which also fixes two bugs in their bamGetTargetCount() function.


> [2] https://github.com/ucscGenomeBrowser/kent/issues/13

On this issue you wrote:

>> tillea commented on 5 Sep 2018
>> I admit htslib is famous for breaking its interface.

Once again: This statement is both rude and false. On the contrary the HTSlib maintainers are careful to minimise breakage to HTSlib interfaces.

Presumably this refers to the various cases of removal of private HTSlib functions that Debian has complained about over the years. The Debian attitude has been that these symbols existed in libhts.so therefore were part of the interface; the HTSlib attitude has been that these functions were never declared in the headers (so they were not callable by user programs) and not documented (so you wouldn't know what to expect them to do anyway) therefore they were not part of the interface. Not acknowledging this distinction is disingenuous.

Debian will be pleased that as of soversion 3, any remaining such private functions no longer appear as visible symbols in libhts.so. So I anticipate that Debian will no longer be accusing htslib of breaking interfaces in this way in the future.

Good luck with packaging blat.

    John




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-packaging/attachments/20200124/8534ff20/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: unbundle-htslib.diff
Type: application/octet-stream
Size: 3284 bytes
Desc: unbundle-htslib.diff
URL: <http://alioth-lists.debian.net/pipermail/debian-med-packaging/attachments/20200124/8534ff20/attachment-0001.obj>