Hi Phil.

Depending on how big your pond is and what its made of... If the base is solid... You can wait till it gets a little bit warmer then just get in there with a net or bowl and scoop out the gravel.

The thing is whether you empty all the water out, or if you get in there when theres water in it... Your still gonna have to scoop the gravel out. So you may as well leave the fish be and keep the water in and get in there yourself..

My pond is about 4.5ft deep, I made the same mistake at one point and thought white gravel would look good at the bottom, over a few months the white became green. So I got in, it was cold and it did take a while.

But on the bright side, you'll only ever have to do it once!