Hi Lleah,
The omega that goes into Equation (2) is the vibrational frequency (or wavenumber of vibration, whatever you want to call it) of the excited/ionized state. You can find it from your fit to the data, because you know that the distance in energy between two peaks is hbar*omega - 2*hbar*omegax where omegax is the anharmonic correction. See Section 2.2 in Stacey's compendium.
As for the sign of dr, I subscribe to Aaron's motivation. When you bring the two atoms in the molecule closer together, you expect the potential to become steeper (intuitively, it should become infinitely steep in the limit r -> 0). The "steepness" is proportional to the force constant k, which is in turn proportional to omega. So if the atoms go closer together, omega should increase wrt the ground state and vice versa.
//
Anders