[sane-devel] Reflecta ProScan / Crystalscan 7200 PIE film scanner update

Thu Aug 23 18:02:11 UTC 2012

Hi Michael,

Good to see you picked it up again!

Regarding your 2nd question: I think we should create a separate backend. I've got two reasons for that: 1) the patch makes the pie backend a very large and complex unit, and 2) I don't want to have to worry about breaking the existing pie functionality every time we modify the usb part. I hope you and Klaus agree.

I did some work on this track, although it is not complete. I took your code (version fall 2011) apart and I am now slowly getting all pieces to work again (in a way I understand). The major thing that does not work yet is sane_read(). Your code is based on the index-format, but I think the pixel-format should be much more practical, since the SANE standard expects RGB pixels to be returned from sane_read().
I divided the code into four sections:
a. Low-level functions which communicate with the device on a byte-level. These implement the usb-wrapper but only handle byte-arrays, no interpretation takes place here. This section is complete and tested.
b. Mid-level scanner functions for scanner commands, which take a meaningful struct as an argument (such as mode_select() which takes a 'struct Mode' as a parameter) and return a 'struct Status' which includes sense data if an error occurs. These functions automatically issue a REQUEST SENSE command if a command returns CHECK SENSE, and repeat a command if the device is reports a busy state. No reason to take care of this anymore at a higher level, which is practical. This section is almost complete, and quite easy to extend. Klaus' device has commands for moving to the next and previous slide, these are not implemented (not yet).
c. Top-level auxiliary functions, e.g. for initialising options, correcting shading, etc. A lot still missing.
d. Top-level SANE interface functions. Missing sane_read() and sane_get_parameters(), the rest implemented according to the procedure of the Windows driver. Not much tested yet.

I see you managed to find and implement dust/scratch removal algorithms. That is very welcome! I hope we can get a positive answer to your third question. It looks like you chose to implement this in the backend, which is practical since it does not break the SANE standard.  This might be the best solution for now.
This makes our wishlist a bit smaller as far as I am concerned. Remains:
- react to a button press (use scanbuttond daemon? it that the solution we want?)
- export infrared data (for other uses than dust/scratch removal; maybe modify the scanimage frontend and stiff.c to allow RGBI TIFF image files to be exported?)

So how do we proceed?
- We have got two pie-patched backends (Klaus' and yours) which I assume should be merged
- We have got a stand-alone backend in an incomplete state (see attachment)
We should decide on a common starting point. If we decide for a separate backend, I would prefer the four-section structure. You and Klaus are familiar with software development, at least more than I am, so we should use that.

Yours,

Jan

-----Original Message-----
From: Michael Rickmann [mailto:mrickma at gwdg.de]
Sent: dinsdag 21 augustus 2012 22:27
To: sane-devel at lists.alioth.debian.org; Vleeshouwers, J.M.
Subject: Reflecta ProScan / Crystalscan 7200 PIE film scanner update

It has become a bit silent about these scanners since Jan Vleeshouwers'
post 8 months ago (
http://lists.alioth.debian.org/pipermail/sane-devel/2011-December/029337.html
). Finally, I had some time to collect what else Jan and I had found out last year and to put it into a patch against the pie backend. Both scanners identify as PIE SF Scanners. They share the same USB id
0x05e3:0x0145 use a SCSI over USB protocol and can be distinguished by byte 0x74 in the INQUIRY return block. Support for the ProScan is quite reasonable now, for the Crystalscan presumably less because calibration (see below) was mostly guessed from what Jan had snooped and communicated. I tested with scanimage and XSane. Currently, Klaus Kaempf seems to work on a PIE MF Scanner ( http://lists.alioth.debian.org/pipermail/sane-devel/2012-July/030092.html ) which is more advanced but presumably not so much different in protocol.
What can the backend do with the SF scanners hardware?
1) scan at resolutions from 20 to 3600 dpi, increment 1
2) read RGB and single pass RGBI data
3) read at 8 and 16 bit [recommended] colour depth
4) calibrate and read in normal and quality [recommended] mode This is what could be detected in snoops using Cyberview and Silverfast under Windows. Silverfast does not support normal mode. Calibration and shading correction is done with the help of software.
What can the backend do by processing the data read?
1) option: clean the infrared colour plane from red spectral overlay
2) option: use the infrared data to remove dirt, output RGB
3) option: simple film grain smoothening (just a triangular blur)
4) option: crop using sanei_magic except for cleaned scans of positive film where an internal routine detects faster and more correctly
5) options: simple colour conversions, constant gamma, invert Scanned positive film really looks good, negative film bad due to still missing colour profiles. Also the options of Xsane for negative film do not improve things. Scanning a 36x24 mm slide at 3600 dpi takes about 88 sec in RGB, 102 sec in RGBI with infrared discarded, 110 sec in RGBI with cleaning and cropping. If one compares these numbers with the ones at http://www.filmscanner.info/en/ReflectaProScan7200.html it seems that the backend handles the scanner correctly.

What can the backend do in spite of the USB additions? Recently I got hold of a 12 year old Adlib JetScan 630 SCSI flatbed scanner which is supported by the pie backend. The scanner still scans at 600 dpi (its
maximum) at “nomal” and “fine” speed; colour, grayscale, halftone and lineart are ok. However at high resolutions and “pro” speed I get hardware “integration time adjustment failure”s “too dark” or “too light”. I get the same errors with SANE-Backends-1.0.22 as included in Ubuntu 12.04 64-bit. I guess, it is the age or a limitation of this kind of scanner. So I think that the direct SCSI part of the backend has not deteriorated.

Description of the added code: I have added three files pie_usb.h contains basic definitions for the USB code sanei_ir.h, sanei_ir.c contain code to remove dirt with help of the infrared plane. I think that this code can migrate to sanei_magic because it may become useful also for other backends. There is still a license problem. I have adapted three threshold routines from M. Emre Celebi's Fourier_0.8 project ( http://sourceforge.net/projects/fourier-ipal/ ) which is licensed under the GNU General Public License Version 2.
Changed files:
sanei_magic.c : sanei_magic_crop can crop images with >8 bit colour depth now pie_scsidefs.h : mainly added support for PIE vendor specific SCSI commands.
pie.c : really a lot. All pure USB functions are prefixed with pie_usb_, all other functions with pie_ (renamed a few) or sane_. The SCSI over USB emulation runs via pie_usb_scsi_wrapper which branches into low level USB code. All other functions talk SCSI. As pie_usb_scsi_wrapper uses the same arguments as sanei_scsi_cmd functions can use a function pointer stored in the scanner structure to do direct SCSI or SCSI over USB. Another possibility to distinguish between the two possibilities is the “model” variable, also in the scanner structure. If it is NULL it indicates a SCSI scanner, otherwise it contains a pointer to the description of an USB scanner. A number of the original functions had become so messed up with if-thens that I wrote pie_usb_ counterparts to ensure that the SCSI part of the backend was not suffering.
A note on calibration:
An outline of what happens during a scan you can find in the introductory comment of pie_usb.h. The best reference for the commands and what they contain is in Jan's post cited above. The scanners use two custom SCSI commands for reading and sending calibration records. During full calibration a record is read from the scanner, then a few calibration lines are read and last, altered calibration data is sent back to the scanner. Although Jan and I could figure out the meaning of most record fields, the snoops did not tell us what was happening between reading and sending in the Windows software. Reverse engineering seemed hopeless. I know Jan would like to have an exact calibration procedure derived from data sheets. There is one huge custom made chip in PIE scanners which presumably does the essential things, and of course there is no documentation. So I started a life-science approach by empirically determining a mathematical model and its parameters. Once one knows what to send for exposure time, gain and offset, one can try different values, scan an empty slide and measure mean color values of the pictures. It melts down to find a relation between exposure time and gain and resulting brightnesses for R, G, B because the Windows programs never changed offset or infrared values between calibration reading and writing. In OpenOffice Calc one can fit a number of curves to the time, gain an illumination values to get a first idea or one can compare the measured values to ones of an own function. In the end the functions giving the best fit were highly non-linear and their parameters had to be determined with help of OpenOffice's evolutionary solver. For a short description of how calibration is done look at the comments before the pie_usb_cali functions in pie.c. If that is not sufficient look at the pie-usb-calibration-formulas document at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/pie_usb-calibration.odt
. By the way, the capability to override calibration values with values read from text files is still in the backend, just look for pie_usb_poke in pie.c.
Shading correction was easy because the scanner tells in the calibration read record what brightness he was aiming at (Jan's Auto-tuned full-scale-level values) for the calibration lines. So, as one knows the outcome one can correct. Calibration lines are always read at full resolution, i.e. every sensor element is used. For a real scan at lower resolution the block of the COPY (0x18) command tells which sensor elements were used so that one can correct each pixel.

I do not have to elaborate the dirt cleaning by infrared here because the pie_usb_sw_post routine in pie.c follows the outline already given in http://lists.alioth.debian.org/pipermail/sane-devel/2011-July/028810.html .

Attached patch includes all this. The patch can be applied to current SANE git 73d450d Sat Aug 18 23:00:14 2012. After a autoreconf [--force] and configure ...[--enable-pthread] it can be build copied and used.
Although I tried to minimize the diff it remains rather unreadable for me. Therefore I used doxygen style comments for the pie USB code. The html output you find at http://wwwuser.gwdg.de/~mrickma/sane-proscan-7200/status-210812/html .
The parent directory of that contains the patch again, the document on calibration, the scripts which were used to process the USB snoops and the doxygen tar ball and a screen shot using XSane.

Now my questions:
Would code like the one in the patch be acceptable in SANE?
Would a separate backend be preferable to patching pie?
What can I do about the license issue already mentioned? I think I could just ask Emre Celebri to allow SANE's license exception for the three routines which were adapted from his GPLed code. He seems to be still at lsus.edu .
Regards
Michael

-------------- next part --------------
A non-text attachment was scrubbed...
Name: SaneReflecta.zip
Type: application/zip
Size: 35234 bytes
Desc: SaneReflecta.zip
URL: <http://lists.alioth.debian.org/pipermail/sane-devel/attachments/20120823/c2461722/attachment-0001.zip>