To remove a current gallery, press the 'Delete selected gallery' button. It is at the upper right of the 'Galleries' panel and looks like a trash can.
After you click on it, a confirmation window will appear. Press 'Yes' if you really want to remove the current gallery. Press 'No' to cancel deletion.
Also, you can remove a gallery, by dragging it with your mouse into the trash can or using the context menu. To do this, right click on a needed gallery and select the 'Remove' item.