We study the effects of galaxy environment on the evolution of the stellar mass function (SMF) over 0.2 < z < 2.0 using the FourStar Galaxy Evolution (ZFOURGE) Survey and NEWFIRM Medium-Band Survey (NMBS) down to the stellar mass completeness limit, logM(*)/M-circle dot > 9.0 (9.5) at z = 1.0 (2.0). We compare the SMFs for quiescent and star-forming galaxies in the highest and lowest environments using a density estimator based on the distance to the galaxies' third-nearest neighbors. For star-forming galaxies, at all redshifts there are only minor differences with environment in the shape of the SMF. For quiescent galaxies, the SMF in the lowest densities shows no evolution with redshift other than an overall increase in number density (phi*) with time. This suggests that the stellar mass dependence of quenching in relatively isolated galaxies both is universal and does not evolve strongly. While at z greater than or similar to 1.5, the SMF of quiescent galaxies is indistinguishable in the highest and lowest densities, at lower redshifts, it shows a rapidly increasing number density of lower-mass galaxies, logM(*)/M-circle dot similar or equal to 9-10, in the highest-density environments. We argue that this evolution can account for all the redshift evolution in the shape of the total quiescent galaxy SMF. This evolution in the quiescent galaxy SMF at higher redshift (z > 1) requires an environmental quenching efficiency that decreases with decreasing stellar mass at 0.5 < z < 1.5 or it would overproduce the number of lower-mass quiescent galaxies in denser environments. This requires a dominant environmental process such as starvation combined with rapid gas depletion and ejection at z > 0.5-1.0 for galaxies in our mass range. The efficiency of this process decreases with redshift, allowing other processes (such as galaxy interactions and ram-pressure stripping) to become more important at later times, z < 0.5.