I dusted my lawn with peat moss...

If the peat moss was a packaged product harvested from the peat bogs of Canada, it is unlikely that it was the cause of the mushrooms. There is a possibility that adding the peat created good sprouting conditions for the mushroom spores that are commonly present everywhere. To begin growing, a colony of the mushrooms must have decomposing matter available, That is usually dead roots, but perhaps in your case, the peat moss gave just enough decomposing matter to encourage the mushrooms. To get rid of mushrooms, rake or mow. Core aerate to loosen up the compacted soil and you will get more air into the soil structure. Eventually the peat moss will break down and the mushroom colony will cease production. Too bad they are not shitake, right?