To surrender, you need to communicate. Usually, ships fighting each other did not engage communication with flags because everyone was at combat posts: it as too dangerous to go up the boat when being fired at. Radios usually were not set on same frequencies between enemies. There were light signal, though.

The problem is that usually, ships fight in a fleet: when they suffer damages, they try to fall back behind their fellowships and they don't surrender on their own.

As a comment said, there is also the fact that weapons were very destructive at each shot compared to the age of sail, and they were fired at a very high distance that reduce the capacity to know if an enemy ship was truly damaged or not.

Now that we say why surrendering was not a common event, let's look about some events of capture in a larger understanding:

• In 1941, Italian cruiser Pola was captured by British sailors: the ship was damaged during the night, its crew left her and later re enter it. A British ship came close and its crew captured the ship with a fast and casulaty-less assault
• Same story with the Altmark, a German (lightly) armed boat captured by a British destroyer off Norway

EDIT:

Apparently some comments need Wikipedia: There were Italian seamen aboard the Pola, and it was not sinking since it needed torpedoes to sink.

After picking up survivors, the destroyers joined Havock and a boarding party was prepared to take Pola, though it was discovered that most of her crew had jumped into the water, and the remaining men were huddled on the forecastle, ready to surrender. Jervis took off the surviving 22 officers and 236 enlisted men from Pola. Wikipedia:Pola

However, post-sail warships were more maneuverable, and had longer-range weapons, which should have made boarding all but impossible.

I'm going to go on a bit about what I think made capturing warships less and less frequent, starting with what I feel is the most important.

Ships are really hard to sink by gunfire

But the enemy is also more maneuverable, and range was only really capitalized on starting in WWI once they could actually hit things.

In my mind, a distinction is the power of the weapons to sink a ship. During age-of-sail, ships fired round solid round shot at each other punching small holes in the hull mostly above the waterline. A frigate would carry 6 and 12 pounders firing shot of 57mm and 120mm. Ships of the line carried 24 and 32 pounders firing 152 mm and 160 mm. Splinter damage did horrendous things to the crew, but without any explosive the small holes which could be patched with canvas and wood plugs.

Age of sail ships could pound away at each other for hours without sinking. Sometimes, as at Trafalgar, they would anchor at point-blank range and blaze away at each other until one surrendered or attempted boarding or both limped away. Very rarely did ships sink in battle. Even in the victory over the Spanish Armada only one ship sank during the battle, four ran aground. Thirty were lost due to bad weather on the return trip.

Rifled breech-loading shell guns

Naval technology stagnated for centuries. A warship of the early 1800s was larger, faster, held more guns, but basically the same as a warship of the 1600s. It had sails, a wooden hull, miles of rope, and a line of muzzle loading smooth-bore cannons mounted broadside firing solid shot.

The introduction of HMS Warrior in 1860, and the later use of ironclads in the US Civil War, put armor firmly on top. But it did not last long.

Three technological innovations made guns much more effective over the span of a couple decades. First, reliable explosive filled shell became available. Now instead of just punching a small hole in the ship, it would explode inside the ship doing horrendous damage.

Rifling became more common, spinning the projectile to make it more accurate at longer ranges.

And breech-loading meant the guns could be loaded much faster.

Guns quickly became more accurate, more powerful, and faster firing. Smaller ships with less guns could do more damage at longer range than a larger age-of-sail ship.

Torpedoes

The other major innovation in sinking ships is the motor torpedo introduced in the late 1800s to the outsized terror of navies. Prior to the motor torpedo, a "torpedo" was any underwater mine. The famous "Damn the torpedoes!" is Admiral Farragut ordering his fleet into a minefield.

A mine was a weapon which could invisibly hit a ship below the waterline, probably unarmored, with a 100 pounds of explosive. Mines were terrifying to warship captains, but you had to run over them. The torpedo is a mine which came to you! What worth was your huge, expensive battleship if a flock of torpedo boats could sink them? (The "torpedo boat destroyer" was invented to deal with them becoming the modern "destroyer").

It turns out hitting a maneuvering warship with torpedoes was much trickier than anyone thought, but they were very useful for a coup de grâce. Rather than having to board a ship to definitively end a battle, one could send a torpedo to finish them off.

Prize Money

One of the motivating factors for sailors in some navies, particularly the British navy, was prize money; money paid to the captain and the crew for capturing ships. Basically state sponsored piracy. Money would be paid directly to the captain and the crew for captured ships. A captain would be highly motivated by prize money to seek boarding action rather than sink enemy vessels. And the crew would go along with it knowing they're to receive a substantial bonus.

Warship specialization

As mentioned earlier, age of sail ships were fundamentally the same. There wasn't all that much difference between a fast merchant ship and a warship, nor between ships of different nations. A captured ship could be crewed, repaired, and sailed to port without much difficulty. Warships were often put back into action, and merchant ships used as small warships. At the height of the Royal Navy's dominance during the Napoleonic Wars the British captured and used so many French vessels is seemed like France was their supplementary shipyard.

While merchant ships of different nations remained largely the same, warship technology diverged, differentiated, and got more complex. Turbines replaced triple-expansion engines. More and more specialized electrical equipment was installed. Guns and ammunition differentiated between nations. And one needed spare parts and expertise to maintain it all. A captured warship became less useful as a warship and more for its intelligence value.

Speed and Radio, Submarines and Aircraft

The final nail, in my opinion, is how fast an enemy could respond to a distress signal. Prior to radio, it would take days or weeks before a battle was reported, and more days and weeks to respond with a warship. This allowed warships to hang around after a battle repairing captured vessels and limping home with them.

In 20th century warfare, with radio and vessels capable of cruising at 20+ knots, the prospect of sitting stopped after a battle restoring a captured vessel was a nightmare. Aircraft and submarines made it worse, at any moment an attack could come out of the sky or from under the sea. The captured vessel was not worth the risk. It might be hastily stripped of its intelligence, and whatever else the crew could grab, and then sunk by torpedo, or more economically, an explosive charge left by the boarding crew.