Here’s a neat technique for creating custom photo edges in Photoshop. (In a nutshell: select the image area, invert, enter quick mask mode, and apply Distortion filters to the mask.) Read more on the topic here and especially here (pdf with illustrations).