How did Japanese small boats win against Mongolian ships?

During the first attempted Mongolian invasion of Japan, fighting mostly occurred on land, just after Mongolian landings. Only local landlords were there to fight, and they did not manage well. In this video we can see that Japanese fought on the beaches, but this is from a movie, probably not that much historically accurate (Mongols are using Naffatun is true, but Mongols were fighting in tight formation).

Hm....this kind of "misunderstanding" or the "lack of the information" occurs mostly due to the language barrier and the lack of the "correct understanding" by so-called "western experts" who can not actually read like "native speakers" the first source in Japanese (or Chinese).

Japanese Wikipedia says,

執権・北条時宗は、このようなモンゴル帝国の襲来の動きに対して以下のような防衛体制を敷いた。

1271年（文永8年・至元8年）、北条時宗は鎮西に所領を持つ東国御家人に鎮西に赴くように命じ、守護の指揮のもと蒙古襲来に備えさせ、さらに鎮西の悪党の鎮圧を命じた[125][126]。当時の御家人は本拠地の所領を中心に遠隔地にも所領を持っている場合があり、そのため、モンゴル帝国が襲来すれば戦場となる鎮西に所領を持つ東国御家人に異国警固をさせることを目的として鎮西への下向を命じたのであった[127]。

Translation

Shikken(English), Tokimune Hojo(English) built the defensive strategy from the prior information he had received that the Mongolian Empire would like to invade Japan sooner or later.[127]

127 「元寇防塁編年史料―注解異国警固番役史料の研究」 in 1971, too hard to translate into English. Simply put "the defensive Japanese whole strategy against the Mongolian invasion".

In 1271(omitted), Tokimune Hojo ordered the Eastern Gokenin(English) who had the feudal lands in the potential landing zone (your "local" area, which is, I think western part of Japan) to prepare the war against Mongols under the command of Shugo(English) as well as quenching the local "ranters"(I have to make an another thread to explain these people, simply put, those who are "out of control of the central government"). The Gokenins (above), had customarily their own feudal lands in western parts of Japan. so (where Mongols were estimated to land) far from their own eastern lands, so that Shikken(above) ordered those eastern Gokenin(above) to go to"their" western parts of Japan for the defense against the invasion against Mongols.

On the Request From The OP.

I see now the OP is more concerned about the naval Japanese attacks on Mongols "ships", so I would like again to quote the Japanese page. (Naturally, inevitably, quite few remained the record because who writes the chronicles of the defeated war of its dynasty.).

Since it is too "complicated", which means there were too many small island basis battles (mostly where war occurred) in the "so called Mongol invasion", I personally would like to take the easiest examples since either way, if I take the minutely detailed records from the multiple sources, I am sorry to say, I might get fatigued and you the OP would likely to have more cases to check and check cursoring the links.

Even introducing this "relatively" minor wars is heavy, I hope you could undersand.

According to the Japanese Wiki,

Let's look at the "tiny"(relatively though) Japanese boats aka ships (I would not like to use this word, since this easy, but I chose to use it for the simplification.)

So as you can see, those ships Japanese used of boats or ships which had the capacity of men from 6-7 men at a boat or ships.

Now, according to the same Japanese Wiki,

1281年（弘安4年・至元18年）、元・高麗軍を主力とした東路軍約40,000～56,989人・軍船900艘と旧南宋軍を主力とした江南軍約100,000人および江南軍水夫（人数不詳）・軍船3,500艘、両軍の合計、約140,000～156,989人および江南軍水夫（人数不詳）・軍船4,400艘の軍が日本に向けて出航した。日本へ派遣された艦隊は史上例をみない世界史上最大規模の艦隊であった[47]。

Translation

In 1281(omitted), the army mainly comprised of Yuan and Goryeo(English), men around 40,000 - 56,989 with the ship of 900 (so 55 men per a ship(around)), and their main strong text force from the south, the area around the former Song Dynasty,, with the forces of infantry of 100,000, and the sailors (the number is unknown), in total army 140,000 - 156,989, on the ships of 4,400 ( so around 34 soldiers per a ship(the number of sailors, according to this page,, 7,000 for 20,500 soldiers, same 7,000 for 33,000 soldiers, which means 7,000 sailors for 20,000 - 50,000 army, which leads to the estimation of the number of sailors were around 30 sailors for each "big" Yuan ship.) This number was the largest so far that attacked Japan in the history which have never seen before. [47]( 47, 「about Tokiyori Hojo and Mongol invasion, by NHK(Japan Broadcasting Association),published in 2001)).

So as you can guess, Japanese forces attacked night and at surprise, Japanese ships employed so called Kobaya which only had 7-10(including a sailors), whereas Monglian ships had around 60 men per a ship, which I think that would hinder the quick manuever of the Yuan ships when attacked by Japanese small forces.

During the second attempt, Japanese failed to land on the same position since the shogunate had built more defenses (both walls and human defenses). Mongolian fleet stood on an island in order to group with the second force of the attack.

Do you mean by the bold part "Mongols failed to land on the same position"?

According to the wikipedia article and this source, Japanese attacked by night, on small, rowing-powered boats, and used them to close Mongoian ships and set them afire.

Ahm, Yes or no. Let's look at a Mongolians source since there were too many "small" battles around the western parts of Japan.(where Mongols landed).

『元史』日本伝によると「冬十月、元軍は日本に入り、これを破った。しかし元軍は整わず、また矢が尽きたため、ただ四境を虜掠して帰還した」[187]としている。

Translation

According to the 「History of Japanese」 in 「Yuan History」, 「October in Winter, the army of Yuan invaded Japan and defeated. But the order of Yuan army was in disorder, and retreated only to capture 4 areas capturing slaves in Japan due to the lack of the numbers of allows」

Overall, from my experience in History SE, even the self-claiming "experts" of Eastern history often mislead readers due to the lack of the understanding either Chinese or Japanese in their native manner. Well, on the contrary, the western version of the Eastern history is more prone to expand in English due to the power of English.