There is no 'one way' to figure out everything. If you know your area, you will soon know where the exits are.
BADIs are usually for adding on customer functionality, not so much for fixes. SAP needs to provide the 'hook' to let you do a BADI in the right place. More than one instance may be called. I find BADIs hard to work with and don't like them much.
User exits are hard 'hooks' (predefined forms), statically linked, ready-to-use and 'empty' (minus some sample code perhaps). They often start with 'userexit' and are therefore easy to find in the main program. Look here first - SAP foresaw where customers should do what in the course of standard processing. Filling fields before calls, doing things before saving, those are tasks you would find here.
Customer functions require 'hooks' from SAP, too. If active, SAP calls function module "EXIT_<REPID>_nnn" where <REPID> is the main program name and 'nnn' the number of the customer function. They are easy to find in SE37. Normally, all you see is a 'ZX*' include that does not yet exist. That's where the code goes. Watch out, sometimes (for example in QM) an include may BE or may NEED to be in many main programs. Make sure you test them all.
Requirements (VOFM) are semi-hardcoded, table-driven, statically compiled and dynamically called. Look for an RxxxxNNN or LxxxxxNNN include. If you create a new requirement, source code is regenerated and linked via the 'NNN' include (literally 'NNN'). What I don't like is that SAP often uses SY-SUBRC as the 'parameter'. Be very careful that a failed or successful command does not alter a yes/no decision here. Work with a local variable instead and set SY-SUBRC at the very end. Make sure you don't exit prematurely. On the bright side, requirements are easy to find and edit in VOFM. Pricing uses 'requirements; that DO something like calculate a base value, but most requirements are yes (SUBRC=0) or no (SUBRC<>0) decisions. The condition technique (pricing, outputs, batch determination, etc.) uses lots of requirements. Be careful when you migrate those: The programs have to be in synch with the (table) data or you may crash. Also, programs have to be regenerated, so you should include R3TR XPRA xxx in the transport (where xxx is the generating program) to execute a regeneration of code upon import. Ask your friendly Basis person about that... ;-)
The second-to-last resort is an implicit enhancement where you can insert code at the beginning and end of a form, function, module, etc. It is still managed by SAP and can therefore survive SPAU easily in case of an update.
The last resort is a true core mod. Try everything else first, but never say never. Some core mods are cleaner than workarounds in the wrong places. Be prepared to defend your proposal to a big and diverse audience. I would suggest that someone with at least 5 years experience looks over it, both to make sure it is necessary and it is well-done.
Regards,
Wolfgang