Finding the modulation

How does one go about finding the correct modulation needed for signal capture and playback?

In my experience, it has just been a game of trial and error. The correct modulation will typically give the highest RSSI on a recording (and will, of course, cause an action on the receiver on replay if the settings are correct). You can typically lookup these settings per device if you can view the FCC ID of the transmitter or receiver