In the days of the wooden sailing ships "gingerbread" referred to the ornate wood carving on the bows and stern of the ship that were covered in gold leaf or "gilded" so to knock the gilt off the gingerbread was a euphemism for something bad happening.