[Debian-science-sagemath] gap: bad interaction between SAGE/atomic_write and GAP/SaveWorkspace [WAS: issue related to compressed manual.six]

Tue Nov 22 05:52:11 UTC 2016

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hello,

I am getting a better picture of what is going on.

On 20/11/16 00:54, Jerome BENOIT wrote:
> 
> 
> On 19/11/16 18:37, Jerome BENOIT wrote:
>> Hello,
> 
>> On 19/11/16 17:43, Bill Allombert wrote:
>>> On Sat, Nov 19, 2016 at 05:18:37AM +0000, Jerome BENOIT wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA512
>>>>
>>>> Hello Forum,
>>>>
>>>> as a matter of fact, discarding the patch <GAP>/debian/patches/fix-compressed-six-files
>>>> fixes the doctests failures related to gap (and libgap).
>>>>
>>>> I have just uploaded un unpatched version of gap at the deb-sci-sage repository:
>>>> please double check !
> 
>>> Hello Jerome,
>>> If I remove fix-compressed-six-files and rebuild the packages, then GAP
>>> is unable to read compressed old-style .six files.
> 
>>> For example install gap-alnuth and do
>>> ?FieldByMatrices
> 
>> Indeed, I can reproduce this issue on two distinct schroots, one with the patch applied and one without it.
> 
> 
> 
>>> With the patch fix-compressed-six-files applied, I get
>>> gap> ?FieldByMatrices
>>> Help: several entries match this topic - type ?2 to get match [2]
> 
>>> [1] Alnuth: FieldByMatrices
>>> [2] Alnuth: FieldByMatricesNC
> 
>>> With the patch removed fix-compressed-six-files, I get
> 
>>> gap> ?FieldByMatrices
>>> Help: no matching entry found
>>> Help: 'FieldByMatrices' is currently undocumented.
>>>       For details, try ?Undocumented Variables
> 
>>> which is wrong.
> 
> What I meant is that I can reproduce this results.
> 
> 
>>> New-style .six files start by a comment lines so are not affected by the
>>> upstream gap bug (which cause the first line to be lost, because rewind
>>> does not work on pipe).
> 
> First the issue on Sage also occurs for the new format (#SIXFORMAT ....).
> In fact, it occurs only when the manual.six is compressed: if, for debugging purpose,
> each installed compressed manual.six is uncompressed, the doctests involving compressed manual.six
> become successful.
> 
> So the issue seems also related to the patch fix-gzip-stringfile which was integrated in the last version of GAP.
> In the header of the patch, we read: `We use pipes for reading gzipped files'.
> What leads me to the question: why we do not use zlib instead ?
> 
> I have also noticed that Sage[Math] works in GAP `-p' mode, what is a bit confusing for me now. 
> 
> 
>>> So it seems to me you need to apply fix-compressed-six-files to libgap
>>> also.
> 
>> The path fix-compressed-six-files does not affect libgap since it patchs only a GAP Include file,
>> more precisely <GAP>/lib/helpbase.gi , while libgap deals with none of them.
>> libgap only deals with the file in <GAP>/src  [1,2]
>> Second, the patches that deal with the material in <GAP>/src are applied [3].
> 
>> [1] http://sources.debian.net/src/libgap-sage/4.8.6%2B3%2B20160327g69a66f0%2Bdsx-1/debian/get-orig-source.sh/
>> [2] http://sources.debian.net/src/libgap-sage/4.8.6%2B3%2B20160327g69a66f0%2Bdsx-1/debian/rules/
>> [3] http://sources.debian.net/src/libgap-sage/4.8.6%2B3%2B20160327g69a66f0%2Bdsx-1/debian/patches/series/ (the two first ones)
> 
>> So now, I think that the patch fix-compressed-six-files might be improved.
> 
>> But before, we must find a way to reproduce in `pure' GAP the issue effectively observed within Sage.

So far I found no way to reproduce is in pure GAP. In fact the patch <GAP>/debian/patches/fix-compressed-six-files is
only evil because it raises an issue which is otherwise eaten/hidden .

Having a look to /tmp/gapsage.log shows that something wrong happens with the GAP function SaveWorkspace within
the atomic_write with-environment. The involved part is implemented in <SAGEMATH>/sage/src/sage/interfaces/gap.py ,
in the method save_workspace of the class Gap .

To get /tmp/gapsage.log , you have to uncomment out the second bottom line of debian/build/usr/share/sage/ext/gap/sage.g .
Grossely this trace file contains the GAP IO.

Currently if you run from <SAGEMATH> the test-command 

$ ( cd sage; ./sage -t -d --long src/sage/interfaces/gap.py )

you get a Failed example. The interesting part is not here but in /tmp/gapsage.log :
up to the sage1 line, all is fine. Then we have an array of random bytes followed by the example failure.
With appropriate GAP Print, we can see that the random bytes are printed by SaveWorkspace within the
atomic_write with-environment implemented in <SAGEMATH>/sage/src/sage/interfaces/gap.py .
Placing the SaveWorkspace outside this atomic_write with-environment (e.g., SaveWorkspace("/tmp/workspace-XXXXXX");)
give a True as GAP output (and we obtained other kind of failures).

I guess that the random bytes in /tmp/gapsage.log are not expected at all, and that they somehow feed some GAP variable.
My current assumption is that the unpatched GAP (i.e., the upstream version) is blind to those random bytes, while the patched
version (the one effectively in Debian unstable) is not, and it is getting confused. Given that the random bytes are certainly a copy
of the workspace, this means that the current sage potentially deals with this large amount of random bytes for nothing but wasting
its time: it could be an important hidden bug.

In short, atomic_write seems to badly interact with SaveWorkspace .

SaveWorkspace is implemented in C in the source file <GAP>/src/saveload.c [1]
atomic_write is part of sagemath [2]

Any hint is welcome to get ride of these random bytes.

[1] http://sources.debian.net/src/gap/4r8p6-1/src/saveload.c/
[2] http://doc.sagemath.org/html/en/reference/misc/sage/misc/temporary_file.html

Thanks,
Jerome
> 
>> Thanks,
>> Jerome
> 
> Thanks,
> Jerome
> 
> 
> 
> 
> 
>>> Cheers,
>>> Bill.
> 
> 
> 
>> _______________________________________________
>> Debian-science-sagemath mailing list
>> Debian-science-sagemath at lists.alioth.debian.org
>> https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath
> 
> 
> 
> _______________________________________________
> Debian-science-sagemath mailing list
> Debian-science-sagemath at lists.alioth.debian.org
> https://lists.alioth.debian.org/mailman/listinfo/debian-science-sagemath
> 

- -- 
Jerome BENOIT, Ph.D. | jgmbenoit-at+rezozer*dot_net
http://www.rezozer.net/

- -- 
Jerome BENOIT | calculus+at-rezozer^dot*net
https://qa.debian.org/developer.php?login=calculus@rezozer.net
AE28 AE15 710D FF1D 87E5  A762 3F92 19A6 7F36 C68B
-----BEGIN PGP SIGNATURE-----

iQQcBAEBCgAGBQJYM90LAAoJED+SGaZ/NsaL3pMgALeUnu5NWzpJ3WoCkBx+j/XM
+mzdBeCTYpwoiCMcrdh1gU9zL5GwbvgF1J13WxhmypBkc/jlvY/K5k3w2rgKPS5C
vSe66HdMEKsHL294znaCkLjnmT8EI1uxbeLVk8ZcCZycaQTD+aQaxr5s/Fp3a+3d
iutrNybXMEThhTpUQCd9ogVO49Gdk/kUOzG92NDHcSqoQnIa+YigXydzHxkXsQ8p
wT7BDKC9d5LB5MMvpaeS8UaZ2Vh713cPkyUN4Tky0fsGM/MW49oaXjz7StrInAuR
RY8ir/kJuorK7knKmG4Co2Sjpbnh9eAfO/cVgyfdm1VzQ5hdef5bzBZIoSri9S8f
pvOUfBUGu4F5TApIlM9XwKijGQh9qVWgWBP8RnEgdu5tYjSimwMkVj/N5UE6gl/E
dTY/hLx/yWKz2UrJbuhkDVo+WYfn2AoaoX6lrlreV8P2kIsZD05aUhCm4BdXeijG
+tb5onA7FhEGqqIOIHLwn8n3nTX/9GVq7P2G3JgOZdki54ZZjkQeS9VO8XDXp0z4
7BvL84Kv4yaWxU8NQ/cAeAiMdQM9GNNMCooKaaKiV0Vv8W7esNKHAA9Qwem0vWSH
nxH4o7IKz6vQl29pe9j3nWC3RL9F8feIfp2bgTAhUBAuY9UZk6buLYB8GyxFDpzQ
DJxguSpGLcMJI7t4rwEuqJrpSp1mbVDUbvYScMsVzumK4Cxar3CzSvr6rVnN0Ilw
lKvohNlUuk++KFww0Vsxs7sFPi2eQjdM8euua1ibNigH6Lc7ES/P/AEMOEyg40Gw
RgvlfU8frZ726TOkd0LlPHe8Mm+g5yTd2iplDZPNzj7Efsx3r45sA1rR/OE37ry7
Q3s8AfSFtvgseDxw4UEqrjNSgxq50E35GdissP1l+Fcmp8FQmcycGj5hUpqrrWfr
PbIaGZNKxfdRbcd57St8HrbfB8otmTd/7h1+g9nAD9Fsdw8uIlTu34bSdl9FKauL
KAN0SC5kwGuCIMt4SlVMn4U+YkO3QHhvH6Kg95RKBrUUUCDxzPPrLCU0iU1bXQPO
M/pBfTo9WzSGxLWusW4I7mNu83EN8BFEnubNOK60img88WrfXpY6bAUsejzyGgiB
CLJOsezhSdcqnj71ASL/mT8Ik6RqbvZNqfxYBMgEraIS35PrbuCwaK+qjyKPVxwC
MJmJ9qzINDzLX5Gi7/w8KzetWrvSqvzx4lUOGMydkZu8xw3w4x32YY1owDQimNUA
umXaZHgfnJgTGyRjzkhW6nV6ee26U/Ck+fNedG7uHkq/9hwp7Hw++m197NFq8Yyy
VMBShjku7aiXPC+XQnCfk1bpjJb6dP6tH0hHEcUIoFQ3EJ8eX1kP5k3GTu5rW0M=
=opeJ
-----END PGP SIGNATURE-----