Scanning from an eSCL Device Using Command Line

eSCL is HP's and Apple's scan protocol. (IETF standards track even?) Uses XML.

xmllint is from the libxml2-utils package.

Get Scanner Status

%  curl -s http://localhost:8080/eSCL/ScannerStatus | xmllint -format -

Get Scanner Capabilities

% curl -s http://localhost:8080/eSCL/ScannerCapabilities | xmllint -format -

Start a scan job

%  curl -v -X POST -d @scansettings.xml  http://localhost:8080/eSCL/ScanJobs

Device should respond with a 201 + Location of the new job. The Location will have a jobid (integer) (or a UUID).

Retrieve the scan job (in this example, 208 is the jobid from the 201 response to the POST)

% curl -s http://localhost:8080/eSCL/ScanJobs/208/NextDocument > out.dat

The 'out.dat' file should be the scanned image. Should be a jpeg or pdf or some other image. Jpeg is most likely.

% file out.dat

out.dat: JPEG image data, JFIF standard 1.01

Simple(ish) scansettings.xml

<?xml version="1.0" encoding="UTF-8"?>
<scan:ScanSettings xmlns:pwg="http://www.pwg.org/schemas/2010/12/sm" xmlns:scan="http://schemas.hp.com/imaging/escl/2011/05/03">
  <pwg:Version>2.0</pwg:Version>
  <pwg:ScanRegions>
    <pwg:ScanRegion>
      <pwg:Height>3300</pwg:Height>
      <pwg:ContentRegionUnits>escl:ThreeHundredthsOfInches</pwg:ContentRegionUnits>
      <pwg:Width>2550</pwg:Width>
      <pwg:XOffset>0</pwg:XOffset>
      <pwg:YOffset>0</pwg:YOffset>
    </pwg:ScanRegion>
  </pwg:ScanRegions>
  <pwg:InputSource>Platen</pwg:InputSource>
  <scan:ColorMode>Grayscale8</scan:ColorMode>
</scan:ScanSettings>

Scanning from Apple AirPrint + AirScan

TL;DR. AirPrint/AirScan with Image Capture + AirScanScanner requires pdl=application/octet-string in mDNS TXT record. Otherwise, AirScanScanner will not work.

AirPrint Requires AirScan.

The Apple AirPrint specification 1.4 requires MFP (Multi-Function Printers; printers with an attached scanner) to also allow scanning over the AirPrint connection. The required protocol is eSCL, an XML over HTTP originally created by HP.

AirScan/eSCL devices advertise themselves over mDNS as _uscan._utcp.  One of the fields in the TXT record is "pdl".

The documentation shows an example: "pdl=application/pdf,image/jpeg"

The AirPrint documentation says PDF and JPEG scanning are required. That's all that's mentioned.  (The pdl field becomes critically important later.)

OSX Image Capture finds network scanners through the mDNS. The actual scanning is done through an executable called AirScanScanner.
/System/Library/Image Capture/Devices/AirScanScanner.app

Only Mavericks can successfully scan from an AirScan/eSCL device. MtnLion connects but fails.

Adding AirScan to Existing AirPrint Device.

I was tasked with adding AirScan support for our scanners.

We have an HP X576 that supports eSCL. Image Capture successfully scans gray/rgb JPEG from HP through eSCL. However, when scanning from my code, Image Capture would only scan rgb. Grayscale would silently fail. No output image.

Only clue was a log message:

Mar 14 16:22:24 latches.local Image Capture[5359]: ImageIO: CGImageSourceCreateWithURL url parameter is nil

The Plot Thickens.

After mimicing as much of the HTTP+XML eSCL as possible, I attacked AirScanScanner with the debugger and dtruss. AirScanScanner writes its temporary images to /var/folders/hw/<longname>/T. The incoming image is written to a temporary file then moved to the user's Pictures folder.

A color scan (for both HP and me) :
rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.awDE7YxI\0", "/Users/davep/Pictures/Scan 93.jpeg\0")         = 0 0

In the case of a grayscale scan, AirScanScanner's behavior deviates. From the HP, AirScanScanner writes a .ica file (??? Image Capture Application intermediate format perhaps?) 

rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.K7RsxSBk\0", "/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/Image Capture_TempScan.LLCnftRz/Scan 91.ica\0")         = 0 0

However, The dtruss traces showed AirScanScanner writing my incoming JPEG image as a TIFF.

rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.jlIExPds\0", "/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/Image Capture_TempScan.N8yDdG7e/Scan 96.tiff\0")         = 0 0

The .tiff is never moved to Pictures. The scan's entire temp directory is never deleted (leaks).

The PDL.

Eventually I found I was advertising pdl slightly differently than the HP. The HP response is shown below:

% dns-sd -L "HP Officejet Pro X576dw MFP [F7816C]" _uscan._tcp
Lookup HP Officejet Pro X576dw MFP [F7816C]._uscan._tcp.local
DATE: ---Fri 21 Mar 2014---
 9:41:34.872  ...STARTING...
 9:41:35.026  HP\032Officejet\032Pro\032X576dw\032MFP\032[F7816C]._uscan._tcp.local. can be reached at HP843497F7816C.local.:8080 (interface 4)
 txtvers=1 vers=2.0 pdl=application/octet-stream,application/pdf,image/jpeg ty=HP\ Officejet\ Pro\ X576dw\ MFP adminurl=http://HP843497F7816C.local. note= UUID=1c852a4d-b800-1f08-abcd-843497f7816c representation= rs=/eSCL cs=binary,color,grayscale is=platen,adf duplex=T

The "pdl=application/octet-stream,application/pdf,image/jpeg" turns out is the key.

When I changed my advertisement to match the HP's, monochrome scanning suddenly began working.

If I changed my advertisement to only "pdl=application/octet-string", scanning still worked.

For reasons unknown, "application/octent-string" is requred in the pdl field of the mDNS advertisement.

EnableLogging and SaveIntermediateFiles.

The AirScanScanner executable contains two very interesting strings: EnableLogging and SaveIntermediate Files.  I would love to be able to enable those debug features. I tried a few tricks with the AirScanScanner.plist but nothing happened. The Image Capture utility launches AirScanScanner so I wasn't able to successfully inject environment variables into it.

In Conclusion.

I'm mostly writing this to help the next firmware engineer tasked with adding Scan support to an AirPrint device. The eSCL is a beautiful simple protocol that simplifies scanning.

Restoring Protected Files on OSX ("chmod operation not permitted")

Work bought me a new MacBookPro last month. The laptop had issues with crashing from the beginning. Finally took it to the Apple Store. They reported they could find no issues and re-imaged the disk.

I've been restoring files from all my backups (DropBox, Super Duper images, and Time Machine image). I've been trying to cherry-pick the restored files using Finder and command line rather than the Time Machine utility. Has been a problem.

When the Apple Store re-imaged the machine, they created a user 'test' to run their diagnostic tools. Once I got the machine back, I added myself 'davep'. The 'test' account was uid 501. The 'davep' account was uid 502. As davep was the first user added before, the Time Machine backups were all owned by 501.

I was unable to chdir into several of the Time Machine disk directories. I couldn't even sudo chmod the trees.

http://superuser.com/questions/326645/sudo-chown-fails-with-operation-not-permitted
http://superuser.com/questions/279235/mac-os-x-why-does-chown-report-operation-not-permitted

The chmod was explicitly forbidden by ACLs on the dirs. Makes sense. OSX wants to protect the backup files as much as possible.

An "ls -led Documents" for example shows:

drwx------+ 36 501  staff  11220 Dec 20 09:48 Documents//
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
 1: group:everyone deny delete

I highlighted the problem id and attribute. The owner is no longer me (I'm 502, the test account was created at 501). The ACL has forbidden anyone from changing the ownership.

After digging through Stackoverflow and Google, I've been using the following commands to clear up the problem:
  
sudo chmod -R -a "everyone deny chown" dirname
sudo chown -R davep dirname

The first chmod will remove the "deny chown" on the tree dirname. The second chown will change the rightful owner ship to me.

Restoring usblp and /dev/usb/lp0

In our firmware development, we use /dev/usb/lp? to talk with our printers. For example:

% python pjltest.py /dev/usb/lp0  # run some PJL tests

Whenever a USB printer is attached, a new char dev node is populated in /dev/usb. The dev node can be read/written just like a file so it's quite easy to communicate with the printer's USB without having to write C code on libusb.

Ubuntu 11.10 did away with the usblp driver. From what I understand, CUPS has stopped using /dev/usb/lp? files to talk with the USB print interface. However, the usblp driver will grab the print interface, effectively blocking CUPS.

The solution in Ubuntu 11.10 (11.04 still had /dev/usb/lp?) was to add usblp to a modeprob blacklist. To restore the /dev/usb/lp?, edit /etc/modprobe.d/blacklist-cups-usblp.conf and comment out the line:

blacklist usblp

For example
#blacklist usblp

On the next USB plug, modprobe is once again free to load usblp.

Note: this will probably break CUPS which is why usblp was added to the blacklist in the first place.

Vess Cola, Part 5.

I had to tinker a while but I think I found a nice way to draw the pieces with ImageMagick's '-draw' operators.

Using the 'floodfill' didn't turn out too well. The line's anti-aliasing caused weird artifacts. (See previous post.) Instead of a square plus two lines, I switched to four triangles.Zooming in looks nicer.

Ha!  Fun.

#!/bin/sh

# draw Vess Cola puzzle bottles
# davep 10-Jul-2011

SIZE=300
HALF=$(( ${SIZE}/2 ))
FOURTH=$(( ${SIZE}/4 ))
EIGHTH=$(( ${SIZE}/8 ))
THREEFOURTH=$(( ${FOURTH}/3 ))

function north {
    if [ $1 == "bottom" ] ; then
        echo "rectangle $(($HALF-$EIGHTH)),0 $(($HALF+$EIGHTH)),$FOURTH"
    else
        echo "polygon $(($HALF-$EIGHTH)),0 $(($HALF+$EIGHTH)),0 $HALF,$FOURTH"
    fi
}

function east {
    if [ $1 == "bottom" ] ; then
        echo "rectangle $(($SIZE-$FOURTH)),$(($HALF-$EIGHTH))  $SIZE,$(($HALF+$EIGHTH))"
    else
        echo "polygon $SIZE,$(($HALF-$EIGHTH)) $SIZE,$(($HALF+$EIGHTH)) $(($SIZE-$FOURTH)),$HALF"
    fi
}

function south {
    if [ $1 == "bottom" ] ; then
        echo "rectangle $(($HALF-$EIGHTH)),$(($SIZE-$FOURTH)) $(($HALF+$EIGHTH)),$SIZE"
    else
        echo "polygon $HALF,$(($SIZE-$FOURTH)) $(($HALF+$EIGHTH)),$SIZE $(($HALF-$EIGHTH)),$SIZE"
    fi
}

function west {
    if [ $1 == "bottom" ] ; then
        echo "rectangle 0,$(($HALF-$EIGHTH)) $FOURTH,$(($HALF+$EIGHTH))"
    else
        echo "polygon 0,$(($HALF-$EIGHTH)) $FOURTH,$HALF 0,$(($HALF+$EIGHTH))"
    fi
}

function piece {
    num=$1
    shift
    convert -size $((${SIZE}+1))x$((${SIZE}+1)) xc:gray -fill white -stroke black \
        -draw "rectangle 0,0 $SIZE,$SIZE" \
        -fill $1 -draw "polygon 0,0 $SIZE,$0 $HALF,$HALF"\
        -fill $3 -draw "polygon $SIZE,0 $SIZE,$SIZE $HALF,$HALF" \
        -fill $5 -draw "polygon 0,$SIZE $SIZE,$SIZE $HALF,$HALF"\
        -fill $7 -draw "polygon 0,0 0,$SIZE $HALF,$HALF"\
        -fill black -draw "$(north $2)"\
        -fill black -draw "$(east $4)"\
        -fill black -draw "$(south $6)"\
        -fill black -draw "$(west $8)"\
        puzzlepiece${num}.png
}

piece 0 red bottom green top blue top yellow bottom
piece 1 red bottom blue bottom green top yellow top
piece 2 red top yellow top blue bottom green bottom
piece 3 green bottom blue bottom green top yellow top
piece 4 yellow bottom red top blue top red bottom
piece 5 green top yellow top red bottom blue bottom
piece 6 red top yellow top green bottom blue bottom
piece 7 blue top green bottom yellow bottom red top
piece 8 red top blue bottom yellow bottom green top

Vess Cola, Part 4.

I started writing a post on how I was arrive at the 4**9 calculation. I wanted a way to illustrate the 4x pattern of the puzzle pieces.

My original drawings were done with OmniGraffle. OmniGraffle is an amazing piece of software. Its ease-of-use gives me something to strive for in my own software.

However, OmniGraffle is quite good at diagramming, drawing squares, shapes, connecting them, etc, but I wanted a nicely colored red/green/yellow/blue square. Flood fill isn't something in OmniGraffle's design.

So how to draw a puzzle piece? I don't want to use my Tkinter GUI app yet (saving that for a future post) but I needed a way to get a simple puzzle piece into a post. And taking a screen shot felt like cheating. Too easy. Yes, I like to find the hard way to do things. I learn a lot that way.

Several years ago, I wrote a little Python+ImageMagick script to do the familiar "Forbidden!" red crossed-out circle. ImageMagick has some very nice drawing primitives. (I'll post the cross.py at some point. I'd like to stay remotely close to the Vess Cola topic for a while.)

First shot was something like:

% convert -size 100x100 xc:gray -fill white -stroke black -draw "rectangle 10,10 90,90" -draw "line 10,10 90,90" -draw "line 90,10 10,90" puzzlepiece.png

Well. It's square with a cross through it. Missing color. Needs some fill. Needs some flood fill. Easy with a paint program. But what's the fun in that? ImageMagick can do flood fill.


% convert -size 100x100 xc:gray -fill white -stroke black \
    -draw "rectangle 10,10 90,90" \
    -draw "line 10,10 90,90" \
    -draw "line 90,10 10,90" \
    -fill blue -draw 'color 20,50 floodfill' \
    -fill red -draw 'color 50,20 floodfill' \
    -fill green -draw 'color 75,50 floodfill' \
    -fill yellow -draw 'color 50,75 floodfill' puzzlepiece2.png

Flood fill is a bit tricky. There can be strange effects around the edges. I'm getting weird white borders. The diagonals are because of anti-aliasing (ImageMagick's web page warns about this). Not sure what's causing the white along the red, green, and blue triangles. Not the yellow, though.

Zooming in on the image via an ImageMagick scaling operation:

% convert -resize 500% -filter Point puzzlepiece2.png p3.png

 

I'm using the 'point' filter to avoid interpolating the result. I didn't want a soft scaled up image. I wanted to see pixel-by-pixel.

(I could have scaled up with OSX's Preview which does a good job of not interpolating pixels. I would have had to take a dreaded screen shot. Nothing against screen shots but I'm having fun looking for a good way to create these images programatically.)

My Tkinter app I used filled triangles. I wonder how that would look in ImageMagick? Could I do it even as an SVG to avoid the jaggies?

Vess Cola, Part 3.

Graphs can be made with pointers or an array configuration.

Speed is of the essence in this application. Why? The sheer number of permutations.

Flexibility isn't important. I don't need the full range of capabilities of a directed graph. Each vertex needs a right and a down edge.

But how many permutations are there? Curious.

There are nine items that can be arranged.

Permutations http://en.wikipedia.org/wiki/Permutation. 9! == 362880

But each card has four positions. Crud. (How did I calculate this again?)  4**9 comes to mind.