[Soc-coordination] Debdelta APT Integration - Report 4

[Ishan Jayawardena]

I continued to work on the rewrite rule and did some modifications to
the existing code. Mainly, I moved out the definitions of the rewrite
rules from code and added it as a new configuration item;
/etc/apt/apt.conf.d/40rewriterules.
Following is the format.
	Aquire::Debdelta::Replace-Rule {
		"http://security.debian.org/"
"http://debdeltas.debian.net/debian-security-deltas/";
		"http://ftp.de.debian.org/debian/"
"http://debdeltas.debian.net/debian-deltas/";
	};
After that, I modified the code to check if each URL has a rewrite
rule. (i) If a rewrite rule is found for that URL, the corresponding
delta URL is constructed and it is queued for downloading. (ii)
Otherwise, the regular deb is queued for downloading. This
modification took longer than I expected because implementing the 2nd
part was difficult and had to try out and test various possibilities
before I finally got it working.

Before this modification, the way I had written it was, I maintained a
default rewrite rule, and replaced every source which did not have a
rewrite rule with that default rule. And then, I constructed debdelta
URLs for each of them and queued them for downloading. If a particular
URL gave a not-found error, I queued the regular deb for downloading.
Since I thought it is a waste of resources and time, I decided to
change the designed as I described above.
	
Then I added experimental support for a 'debdelta' line in the
sources.list file. But, we could not use it for any better purpose. As
I believe, we can make use of it only we are going to indexes indexes
for deltas. Michael suggested two possible usages for the debdelta
line. One is to use it to enable/disable fetching deltas for a given
source. The other suggestion is to use it to provide URLs that for the
rewrite rule. In both these cases, we will still have to define
rewrite rules explicitly to map sources to debdelta sources. But since
we can achieve both these easily with the above config file I
described, I did not work on them.

On the other hand, as I think, if we can store the deltas in the same
place as the debs, we can totally get rid of rewrite rules. Also, if
we can support indexes for debdeltas, we will be able to use the
debdelta entries of sources.list to get details about the deltas and
use them in download progress reporting/verifying/patching stages etc.

I also managed to find a solution to the problem of redirecting the
stdout of the debdelta method. I used the dup() system call together
with ExecFork() function to invoke the debpatch program by creating a
pipe. Then I collected the output of debpatch program to a single
string which can later be used in the debdelta method.

Michael suggests that we add a --status-fd option to the debpatch
program similar to that of dpkg and then use it to pass a file
descriptor from apt, which debpatch can use to send machine readable
status data(which apt can parse) back to apt. But for this, we will
have to do a considerable modification to debpatch's existing progress
reporting method.

But again I have some problems with this approach. The first one is,
we are invoking debpatch from within debdelta, which is another method
invoked by apt. But dpkg is not invoked as a method by apt. Therefore,
apt can directly communicate with dpkg and sending an --status-fd
option and manipulating its content is easier. But if we some how
manage to support --status-fd within the debdelta method, we will have
to send those data back to apt. As I think, apt invokes dpkg after
completing all the downloads and when dpkg runs, there is no any
download related progress data coming from apt. But the situation is
different with debdelta. i.e. debdelta method is invoked for each
successful debdelta and it is run in parallel with the downloads. So
we cannot mix the two types of progress details. Michael suggested to
show the progress of debpatch method only for the remaining deltas at
the point when all the downloads are finished. As I understand, for
that, I must close/open the file descriptor of debdelta method
conditionally within apt. Currently I am looking into this because I
do not have a clear idea about how to implement that.

If you have any suggestions please let me know.

My APT branch in Launchpad: https://code.launchpad.net/~udeshike/apt/apt-ishan
-- 
Regards,
Ishan Jayawardena.