It's not as easy to set up as clicking a checkbox like Ubuntu/Gnome, but it is possible to type proper Esperanto characters in Windows using Right Alt as a modifier key. You need to create and install an alternate keyboard layout and then set the new layout active.
The program Keyboard Layout Creator is used to create the layout, and is available for free from Microsoft. Once it's downloaded and installed, start the program. Navigate File > Load Existing Keyboard and then select your primary keyboard layout (standard US layout in my case). You'll use this as a base and augment it with the Esperanto characters.
For each key that will should an accented character, right-click its position on the virtual keyboard and click "Properties for VK_? in all shift states". A dialog will appear in which the necessary Unicode code points can be entered.
The code points for the accented Esperanto letters are shown below, as well as for the Euro and Spesmilo just for fun:
If you don't want to enter the Unicode values yourself, feel free to use a copy of my keyboard definition file.
When you're finished setting the code points for each letter, navigate Project > Test Keyboard Layout to test them. Then, navigate Project > Properties to provide the necessary name and other descriptive information for the new layout. The name cannot be longer than eight characters, so I simply named mine "EO".
Once you're satisfied with the layout, navigate Project > Build DLL and Setup Package. The keyboard layout will be compiled to a binary format usable by Windows and be saved to your hard drive. Run the setup.exe installer that was written to disk install the layout. The installer will detect your system's architecture and launch the appropriate sub-installer.
Restart your computer once the installer is finished. You'll then be able to toggle between your original layout and the Esperanto layout using the Language Bar.
I set the augmented layout as my default keyboard layout (although I don't recommend this unless you're computer savvy). To do this on Windows 7, go to the Start menu, type "language" in the search bar, and select "Change keyboard and input methods". Click the "Change keyboards" button and you'll see the Text Services and Input Languages dialog. Under the General tab, set the new layout as the default input language and remove the entry for your original layout in the installed services tree.
On Windows 8, start typing "language" on the Start screen and then select "Change input methods" from the Settings group.
The Windows 8 Language panel more or less provides the same functionality as its Windows 7 counterpart but with a less user-friendly manner. The Input method is accessible through the options link.