At first pass, this seems like a fairly simple problem: How do you zoom to mouse position in Core Animation? I suppose looking at it now, once I've figured it, out it seems quite straightforward.
The problem lies in the fact that I had a complicated layer structure last time I tried to do this. The first time I tried to arrange the layer hierarchy for my sequential image viewing program, I had far too many layers. I had a root layer, then there was a zoom layer, which would theoretically just zoom. Then there was a layer for panning. Finally there was a layer encasing the images, and then the images themselves. This turned out to be a housekeeping nightmare.
The simple solution is to use two layers. The required root layer, and a layer to contain the images. The image layout is done with respect to the images layer. The panning of the images is handled by changing the position of the images layer. The zooming is handled by applying a scaling transform to the images layer. Simple!
There's still a trick to get things to work just the way I wanted. In order to have the layer zoom in and out while centered on the mouse cursor, I redefine the scroll wheel function for the containing window. Here I figure out the coordinates of the mouse point in the layer being zoomed. It does not matter if the coordinates are outside the layer. I then create a new anchor point by dividing the coordinates I just figured out by the width and height of the layer. This gives me a unit length, rather than a pixel length (damn Apple for making things this complicated; I still don't see why they need to have two different coordinate schemes). Finally, I save the frame of the layer I want to zoom, set up a transaction without actions, set the new anchor point, reset the old frame, and end the transaction. I have to save the frame because changing the anchor point moves the layer, so to change the anchor point in place I reset the frame back to what it was. Now I can apply the scaling transform to the layer however I wish and move on with my life.
I wish I had a guide like this back when I was first trying to solve this problem half a year ago, so maybe this will help somebody else. Know a simpler way to do it? Drop me a line!