[sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update

Tue Aug 21 20:27:06 UTC 2012

It has become a bit silent about these scanners since Jan Vleeshouwers' 
post 8 months ago ( 
http://lists.alioth.debian.org/pipermail/sane-devel/2011-December/029337.html 
). Finally, I had some time to collect what else Jan and I had found out 
last year and to put it into a patch against the pie backend. Both 
scanners identify as PIE SF Scanners. They share the same USB id 
0x05e3:0x0145 use a SCSI over USB protocol and can be distinguished by 
byte 0x74 in the INQUIRY return block. Support for the ProScan is quite 
reasonable now, for the Crystalscan presumably less because calibration 
(see below) was mostly guessed from what Jan had snooped and 
communicated. I tested with scanimage and XSane. Currently, Klaus Kaempf 
seems to work on a PIE MF Scanner ( 
http://lists.alioth.debian.org/pipermail/sane-devel/2012-July/030092.html ) 
which is more advanced but presumably not so much different in protocol.
What can the backend do with the SF scanners hardware?
1) scan at resolutions from 20 to 3600 dpi, increment 1
2) read RGB and single pass RGBI data
3) read at 8 and 16 bit [recommended] colour depth
4) calibrate and read in normal and quality [recommended] mode
This is what could be detected in snoops using Cyberview and Silverfast 
under Windows. Silverfast does not support normal mode. Calibration and 
shading correction is done with the help of software.
What can the backend do by processing the data read?
1) option: clean the infrared colour plane from red spectral overlay
2) option: use the infrared data to remove dirt, output RGB
3) option: simple film grain smoothening (just a triangular blur)
4) option: crop using sanei_magic except for cleaned scans of positive 
film where an internal routine detects faster and more correctly
5) options: simple colour conversions, constant gamma, invert
Scanned positive film really looks good, negative film bad due to still 
missing colour profiles. Also the options of Xsane for negative film do 
not improve things. Scanning a 36x24 mm slide at 3600 dpi takes about 88 
sec in RGB, 102 sec in RGBI with infrared discarded, 110 sec in RGBI 
with cleaning and cropping. If one compares these numbers with the ones 
at http://www.filmscanner.info/en/ReflectaProScan7200.html it seems that 
the backend handles the scanner correctly.

What can the backend do in spite of the USB additions? Recently I got 
hold of a 12 year old Adlib JetScan 630 SCSI flatbed scanner which is 
supported by the pie backend. The scanner still scans at 600 dpi (its 
maximum) at “nomal” and “fine” speed; colour, grayscale, halftone and 
lineart are ok. However at high resolutions and “pro” speed I get 
hardware “integration time adjustment failure”s “too dark” or “too 
light”. I get the same errors with SANE-Backends-1.0.22 as included in 
Ubuntu 12.04 64-bit. I guess, it is the age or a limitation of this kind 
of scanner. So I think that the direct SCSI part of the backend has not 
deteriorated.

Description of the added code: I have added three files
pie_usb.h contains basic definitions for the USB code
sanei_ir.h, sanei_ir.c contain code to remove dirt with help of the 
infrared plane. I think that this code can migrate to sanei_magic 
because it may become useful also for other backends. There is still a 
license problem. I have adapted three threshold routines from M. Emre 
Celebi's Fourier_0.8 project ( 
http://sourceforge.net/projects/fourier-ipal/ ) which is licensed under 
the GNU General Public License Version 2.
Changed files:
sanei_magic.c : sanei_magic_crop can crop images with >8 bit colour 
depth now
pie_scsidefs.h : mainly added support for PIE vendor specific SCSI commands.
pie.c : really a lot. All pure USB functions are prefixed with pie_usb_, 
all other functions with pie_ (renamed a few) or sane_. The SCSI over 
USB emulation runs via pie_usb_scsi_wrapper which branches into low 
level USB code. All other functions talk SCSI. As pie_usb_scsi_wrapper 
uses the same arguments as sanei_scsi_cmd functions can use a function 
pointer stored in the scanner structure to do direct SCSI or SCSI over 
USB. Another possibility to distinguish between the two possibilities is 
the “model” variable, also in the scanner structure. If it is NULL it 
indicates a SCSI scanner, otherwise it contains a pointer to the 
description of an USB scanner. A number of the original functions had 
become so messed up with if-thens that I wrote pie_usb_ counterparts to 
ensure that the SCSI part of the backend was not suffering.
A note on calibration:
An outline of what happens during a scan you can find in the 
introductory comment of pie_usb.h. The best reference for the commands 
and what they contain is in Jan's post cited above. The scanners use two 
custom SCSI commands for reading and sending calibration records. During 
full calibration a record is read from the scanner, then a few 
calibration lines are read and last, altered calibration data is sent 
back to the scanner. Although Jan and I could figure out the meaning of 
most record fields, the snoops did not tell us what was happening 
between reading and sending in the Windows software. Reverse engineering 
seemed hopeless. I know Jan would like to have an exact calibration 
procedure derived from data sheets. There is one huge custom made chip 
in PIE scanners which presumably does the essential things, and of 
course there is no documentation. So I started a life-science approach 
by empirically determining a mathematical model and its parameters. Once 
one knows what to send for exposure time, gain and offset, one can try 
different values, scan an empty slide and measure mean color values of 
the pictures. It melts down to find a relation between exposure time and 
gain and resulting brightnesses for R, G, B because the Windows programs 
never changed offset or infrared values between calibration reading and 
writing. In OpenOffice Calc one can fit a number of curves to the time, 
gain an illumination values to get a first idea or one can compare the 
measured values to ones of an own function. In the end the functions 
giving the best fit were highly non-linear and their parameters had to 
be determined with help of OpenOffice's evolutionary solver. For a short 
description of how calibration is done look at the comments before the 
pie_usb_cali functions in pie.c. If that is not sufficient look at the 
pie-usb-calibration-formulas document at 
http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/pie_usb-calibration.odt 
. By the way, the capability to override calibration values with values 
read from text files is still in the backend, just look for pie_usb_poke 
in pie.c.
Shading correction was easy because the scanner tells in the calibration 
read record what brightness he was aiming at (Jan's Auto-tuned 
full-scale-level values) for the calibration lines. So, as one knows the 
outcome one can correct. Calibration lines are always read at full 
resolution, i.e. every sensor element is used. For a real scan at lower 
resolution the block of the COPY (0x18) command tells which sensor 
elements were used so that one can correct each pixel.

I do not have to elaborate the dirt cleaning by infrared here because 
the pie_usb_sw_post routine in pie.c follows the outline already given 
in 
http://lists.alioth.debian.org/pipermail/sane-devel/2011-July/028810.html .

Attached patch includes all this. The patch can be applied to current 
SANE git 73d450d Sat Aug 18 23:00:14 2012. After a autoreconf [--force] 
and configure ...[--enable-pthread] it can be build copied and used. 
Although I tried to minimize the diff it remains rather unreadable for 
me. Therefore I used doxygen style comments for the pie USB code. The 
html output you find at 
http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/html . 
The parent directory of that contains the patch again, the document on 
calibration, the scripts which were used to process the USB snoops and 
the doxygen tar ball and a screen shot using XSane.

Now my questions:
Would code like the one in the patch be acceptable in SANE?
Would a separate backend be preferable to patching pie?
What can I do about the license issue already mentioned? I think I could 
just ask Emre Celebri to allow SANE's license exception for the three 
routines which were adapted from his GPLed code. He seems to be still at 
lsus.edu .
Regards
Michael

-------------- next part --------------
A non-text attachment was scrubbed...
Name: piuspatch.diff.gz
Type: application/x-gzip
Size: 57839 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/sane-devel/attachments/20120821/b1f779e9/attachment-0001.bin>