When testing a custom action, it is usally practical to have the CA write some information to the installer log, so that you can verify that the CA has been called properly. But what if the installer log does not show up your information but instead the installation aborts and the log only states that the custom action returned 3 (ERROR_INSTALL_FAILURE)?
There may be various reasons for this to happen, but the most common reason is probably that your DLL was unable to load because of unsatisfied dependencies. Note that for the CA to work on all machines, it may only have dependencies to ‘standard DLLs’ — those DLLs that are available on a fresh Windows install.
Odds are that if you created the DLL with Visual C++ 2005 using standard settings (i.e. compiling with /MD[d]), VC has introduced a dependency to msvcr80.dll. As msvcr80.dll is not a ‘standard Windows DLL’, this will result in a DLL loading failure. The solution in this case is mostly as easy as switching to /MT[d].