Merge Git Repo to New Git Rep.

Here's the problem I needed to solve: a group needed their own Git area to work in. We wanted to be able to work in our own Git trees without breaking each other's tree.

We are running gitolite on our corporate Git servers. We can't add repos without going through an administrator.

We have an automated build server that fetches the entire Git repo. The more branches in the repo, the longer it takes to fetch and the longer each automated build takes.

The solution was to create a new set of repos, parallel to the original "main" repos. (Our gitolite admin created those repos for us.) Now, how to get the current code from the main repo to the second group's new repos?

https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history

# get the new repo (currently empty)
git clone git@gitolite:new-group-dev/reponame reponame

# should now have an empty tree called "reponame"
# cd reponame

# add the secondary repo
git remote add r1remote git@gitolite:old-group-dev/other-repo-name

# fetch the secondary repo code
git fetch r1remote

# and at this point the command diverge from the Stack Overflow. I'm pulling in the branch I created
# that the other group is going to start from.
git merge r1remote/b/dpoole/davesbranch

# all my code should now be in the current directory

# do NOT 'git remote rm' the r1remote; we want to be able to merge changes from it

# push code to new-group-dev/reponame on the gitolite server
git push origin master

# we should have two remotes.
% git remote -v
origin    git@gitolite:new-group-dev/reponame (fetch)
origin    git@gitolite:new-group-dev/reponame (push)
r1remote    git@gitolite:old-group-dev/other-repo-name (fetch)
r1remote    git@gitolite:old-group-dev/other-repo-name (push)

# TODO update this post when we start cross merging. That should be fun!

Adding CUPS to Raspberry Pi

In order to use localhost:631 to manage CUPS, the pi user needs to be in the lp and lpadmin groups.

% sudo usernmod -a -G lp,lpadmin pi

Probably need to logout/login again.

The CUPS web login will prompt user username/password. Use the pi user and your pi password.

Adding New Scanner to Raspberry Pi

More notes on getting scan working on Raspberry Pi running NOOBs (Debian).

A"permission denied" problem is usually the first stumbling block using libusb to directly talk to scanner.  SANE will automatically configure the scanner to be read/writeable by the user.

SANE (http://www.sane-project.org/) is awesome sauce. But it doesn't support the weird scanners I get to handle.

To add a new scanner, add the vid/pid to /lib/udev/rules.d/60-libsane.rules after LABEL="libsane_usb_rules_begin" (towards start of file) and before LABEL="libsane_rules_end".

SANE's udev rules finish by calling /bin/setfacl to set permissions on the USB devnode. Just adding the scanner's vid/pid to the sane udev rules used to be enough (Ubuntu 12.04). Not so anymore.

The Raspberry Pi Debian has a "scanner" group. The scanner dev node is created rw on that group. So I needed to add the user 'pi' to the scanner group with usermod.

I don't know if the scanner group is a new feature of SANE or something in the newer Debians. (I'm running an older Ubuntu on my laptop). But I can now talk to my scanner without being root.

Changing Console Keyboard on Raspberry Pi

I'm ramping up on using Raspberry Pi as a scanner development machine. As such, I'm installing all the usual suspects: sane, libusb, PIL, matplotlib, and numpy.

The default console keyboard isn't "us". As a US keyboard user, I found things like | (pipe) weren't working for me.

The console keyboard is configured in /etc/default/keyboard.  See also keyboard(5).  I changed KBDLAYOUT="gb" to KBDLAYOUT="us" and rebooted. Voila! My keyboard now works.

Scanning from an eSCL Device Using Command Line

eSCL is HP's and Apple's scan protocol. (IETF standards track even?) Uses XML.

xmllint is from the libxml2-utils package.

Get Scanner Status

%  curl -s http://localhost:8080/eSCL/ScannerStatus | xmllint -format -

Get Scanner Capabilities

% curl -s http://localhost:8080/eSCL/ScannerCapabilities | xmllint -format -

Start a scan job

%  curl -v -X POST -d @scansettings.xml  http://localhost:8080/eSCL/ScanJobs

Device should respond with a 201 + Location of the new job. The Location will have a jobid (integer) (or a UUID).

Retrieve the scan job (in this example, 208 is the jobid from the 201 response to the POST)

% curl -s http://localhost:8080/eSCL/ScanJobs/208/NextDocument > out.dat

The 'out.dat' file should be the scanned image. Should be a jpeg or pdf or some other image. Jpeg is most likely.

% file out.dat

out.dat: JPEG image data, JFIF standard 1.01

Simple(ish) scansettings.xml

<?xml version="1.0" encoding="UTF-8"?>
<scan:ScanSettings xmlns:pwg="http://www.pwg.org/schemas/2010/12/sm" xmlns:scan="http://schemas.hp.com/imaging/escl/2011/05/03">
  <pwg:Version>2.0</pwg:Version>
  <pwg:ScanRegions>
    <pwg:ScanRegion>
      <pwg:Height>3300</pwg:Height>
      <pwg:ContentRegionUnits>escl:ThreeHundredthsOfInches</pwg:ContentRegionUnits>
      <pwg:Width>2550</pwg:Width>
      <pwg:XOffset>0</pwg:XOffset>
      <pwg:YOffset>0</pwg:YOffset>
    </pwg:ScanRegion>
  </pwg:ScanRegions>
  <pwg:InputSource>Platen</pwg:InputSource>
  <scan:ColorMode>Grayscale8</scan:ColorMode>
</scan:ScanSettings>

Scanning from Apple AirPrint + AirScan

TL;DR. AirPrint/AirScan with Image Capture + AirScanScanner requires pdl=application/octet-string in mDNS TXT record. Otherwise, AirScanScanner will not work.

AirPrint Requires AirScan.

The Apple AirPrint specification 1.4 requires MFP (Multi-Function Printers; printers with an attached scanner) to also allow scanning over the AirPrint connection. The required protocol is eSCL, an XML over HTTP originally created by HP.

AirScan/eSCL devices advertise themselves over mDNS as _uscan._utcp.  One of the fields in the TXT record is "pdl".

The documentation shows an example: "pdl=application/pdf,image/jpeg"

The AirPrint documentation says PDF and JPEG scanning are required. That's all that's mentioned.  (The pdl field becomes critically important later.)

OSX Image Capture finds network scanners through the mDNS. The actual scanning is done through an executable called AirScanScanner.
/System/Library/Image Capture/Devices/AirScanScanner.app

Only Mavericks can successfully scan from an AirScan/eSCL device. MtnLion connects but fails.

Adding AirScan to Existing AirPrint Device.

I was tasked with adding AirScan support for our scanners.

We have an HP X576 that supports eSCL. Image Capture successfully scans gray/rgb JPEG from HP through eSCL. However, when scanning from my code, Image Capture would only scan rgb. Grayscale would silently fail. No output image.

Only clue was a log message:

Mar 14 16:22:24 latches.local Image Capture[5359]: ImageIO: CGImageSourceCreateWithURL url parameter is nil

The Plot Thickens.

After mimicing as much of the HTTP+XML eSCL as possible, I attacked AirScanScanner with the debugger and dtruss. AirScanScanner writes its temporary images to /var/folders/hw/<longname>/T. The incoming image is written to a temporary file then moved to the user's Pictures folder.

A color scan (for both HP and me) :
rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.awDE7YxI\0", "/Users/davep/Pictures/Scan 93.jpeg\0")         = 0 0

In the case of a grayscale scan, AirScanScanner's behavior deviates. From the HP, AirScanScanner writes a .ica file (??? Image Capture Application intermediate format perhaps?) 

rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.K7RsxSBk\0", "/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/Image Capture_TempScan.LLCnftRz/Scan 91.ica\0")         = 0 0

However, The dtruss traces showed AirScanScanner writing my incoming JPEG image as a TIFF.

rename("/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/temp.jlIExPds\0", "/var/folders/hw/vb001gqj3zv8vrlbr4tn8cqw0000gp/T/Image Capture_TempScan.N8yDdG7e/Scan 96.tiff\0")         = 0 0

The .tiff is never moved to Pictures. The scan's entire temp directory is never deleted (leaks).

The PDL.

Eventually I found I was advertising pdl slightly differently than the HP. The HP response is shown below:

% dns-sd -L "HP Officejet Pro X576dw MFP [F7816C]" _uscan._tcp
Lookup HP Officejet Pro X576dw MFP [F7816C]._uscan._tcp.local
DATE: ---Fri 21 Mar 2014---
 9:41:34.872  ...STARTING...
 9:41:35.026  HP\032Officejet\032Pro\032X576dw\032MFP\032[F7816C]._uscan._tcp.local. can be reached at HP843497F7816C.local.:8080 (interface 4)
 txtvers=1 vers=2.0 pdl=application/octet-stream,application/pdf,image/jpeg ty=HP\ Officejet\ Pro\ X576dw\ MFP adminurl=http://HP843497F7816C.local. note= UUID=1c852a4d-b800-1f08-abcd-843497f7816c representation= rs=/eSCL cs=binary,color,grayscale is=platen,adf duplex=T

The "pdl=application/octet-stream,application/pdf,image/jpeg" turns out is the key.

When I changed my advertisement to match the HP's, monochrome scanning suddenly began working.

If I changed my advertisement to only "pdl=application/octet-string", scanning still worked.

For reasons unknown, "application/octent-string" is requred in the pdl field of the mDNS advertisement.

EnableLogging and SaveIntermediateFiles.

The AirScanScanner executable contains two very interesting strings: EnableLogging and SaveIntermediate Files.  I would love to be able to enable those debug features. I tried a few tricks with the AirScanScanner.plist but nothing happened. The Image Capture utility launches AirScanScanner so I wasn't able to successfully inject environment variables into it.

In Conclusion.

I'm mostly writing this to help the next firmware engineer tasked with adding Scan support to an AirPrint device. The eSCL is a beautiful simple protocol that simplifies scanning.

Restoring Protected Files on OSX ("chmod operation not permitted")

Work bought me a new MacBookPro last month. The laptop had issues with crashing from the beginning. Finally took it to the Apple Store. They reported they could find no issues and re-imaged the disk.

I've been restoring files from all my backups (DropBox, Super Duper images, and Time Machine image). I've been trying to cherry-pick the restored files using Finder and command line rather than the Time Machine utility. Has been a problem.

When the Apple Store re-imaged the machine, they created a user 'test' to run their diagnostic tools. Once I got the machine back, I added myself 'davep'. The 'test' account was uid 501. The 'davep' account was uid 502. As davep was the first user added before, the Time Machine backups were all owned by 501.

I was unable to chdir into several of the Time Machine disk directories. I couldn't even sudo chmod the trees.

http://superuser.com/questions/326645/sudo-chown-fails-with-operation-not-permitted
http://superuser.com/questions/279235/mac-os-x-why-does-chown-report-operation-not-permitted

The chmod was explicitly forbidden by ACLs on the dirs. Makes sense. OSX wants to protect the backup files as much as possible.

An "ls -led Documents" for example shows:

drwx------+ 36 501  staff  11220 Dec 20 09:48 Documents//
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
 1: group:everyone deny delete

I highlighted the problem id and attribute. The owner is no longer me (I'm 502, the test account was created at 501). The ACL has forbidden anyone from changing the ownership.

After digging through Stackoverflow and Google, I've been using the following commands to clear up the problem:
  
sudo chmod -R -a "everyone deny chown" dirname
sudo chown -R davep dirname

The first chmod will remove the "deny chown" on the tree dirname. The second chown will change the rightful owner ship to me.