Both sub-images and setSpriteAnimation() can be used to access multiple small images held within a larger image, allowing multiple sprites to share a common image source.
Subimages method requires a simulated load command for each frame, setSpriteAnimation() accesses the images using frame numbers based on an imagined grid within the larger image.
The other difference is to do with the size of the images and their placement.
setSpriteAnimation() accesses the frames based on a uniform grid and so requires the smaller images to be on boundaries equivalent to the image size - a 64x64 frame used in this way needs to lie on a 64x64 boundary in the larger image.
Different sized frames can be used providing each lies on a boundary which matches it's size - so 64x64 images can be used in the same image as 64x128 , 32x32 , 16x256, etc. Providing the large image can be subdivided into a grid of those sizes, and the frames sit on equivalent boundaries.
The subimage file specifies the position of the smaller images, allowing frames of varying sizes and (more importantly) frames which cannot be accessed using the grid style approach of setSpriteAnimation().
SO in short.
Tile sets, fixed width fonts and frames of consistent size - use setSpriteAnimation()
Variable width fonts and Frames of varying sizes or on non-boundary positions - use subimages.
If you think you've got that and want more, take a look at how they can be used
at the same time to get around odd sizes;
http://forum.thegamecreators.com/?m=forum_view&t=195871&b=41
eg. having a 50 x 50 tile size and using set spriteanimation() to access it on a 1024x1024 image (normally not possible)