Why was the Red Army successful against the Japanese in 1939 but perform poorly against Finland 1939-40 and early part of WW2?

Terrain and an arguably lesser extent of the purges

First, of all, it must be noted that both in the case of Khalkin Gol and the Winter War, the Soviet Union did win, i.e. the Kremlin managed to force its political will on its opponents. What remains to be seen is the price of these victories, relative to the strength of their opponents. So let's compare them :

The first thing to note is the big difference in size and population: Finland was a small country with an estimated population of 3.5 million during the Winter War; Japan's population was over 70 million. Japan, of course, had much larger armed forces (supplied by a well-developed industry), but it did not use all of its strength against the USSR (and vice versa). Finland, on the other hand, mobilized practically its whole available male population for the war.

The Japanese air force was a sizeable opponent to the Soviets compared to the Finns, and according to various estimates they may even had superiority early in the war. Nevertheless, by August, VVS was able to wrestle control of the air and give significant close support in unfolding battles. It must be said that clear summer weather and open ground on the Mongolian steppes gave almost perfect terrain for the employment of air power.

On the other hand, the Finnish air force was clearly numerically inferior. Although they claimed many air victories, the reality is somewhat different: both VVS and Baltic Fleet aviation lost less than 150 aircraft, most of them in accidents. Furthermore, the Soviets managed to fly more then 100 000 combat sorties, dropped thousands of bombs etc ...

What really hampered the effectiveness of Soviet aviation was simply weather and terrain. The Winter War happened in the winter, with reduced visibility and over snow-covered forests. In such circumstances, with the limited radio and navigation equipment Soviets had, it was difficult to find and hit Finnish targets with any precision. Soviet aviation could not give direct air support in Finland comparable to what they could give against the Japanese at Khalkin Gol. Instead, they were either blindly bombing supposed Finnish positions, or attempting a strategic campaign against (static) targets in rear areas. Arguably, this had little effect on the outcome of the war.

What about ground forces ? The Japanese had a few dozen of both Type 89 I-Go and Type 95 Ha-Go tanks. Soviets primarily relied on BT-5 and BT-7. BT tanks were lightly armored and had no advantage in that regard compared to Japanese tanks, but were much faster and had a better main gun. The Biggest advantage was numbers - the Soviets fielded 400-500 of these tanks, plus various armored cars. Coupled with better motorization of infantry and artillery (thousands of trucks, tank riding troops), the Soviets were able to conduct mobile operations in accordance with their deep battle doctrine.

Thus, despite the lack of coordination between various units and branches of military, at Khalkin Gol we had essentially a WW2 army (Red Army) fighting basically a WW1 army (Japanese army).

It was a completely different situation in Finland. The Finns practically had no tanks and very few pieces of heavy artillery. But in snowy, forested terrain this proved to be an advantage. The Main Soviet tank used in that theatre was the T-26, although other newer designs did not prove themselves to be much better. They simply had to stay on or near roads, which Finns used for ambushes, setting up mine fields, cutting off supplies etc ... The Soviet infantry (often from Ukrainian regions, unaccustomed to Finnish winter forests) and artillery were also confined to the roads, subject to Finnish motti tactics, since the Finns often used highly mobile ski-troops to rapidly move from place to place and isolate them. Overall, in the Finnish theatre, the Soviets were forced to revert to WW1 slow and deliberate firepower tactics, which finally did yield somewhat favorable results in February of 1940, forcing the Finns to finally accept Soviet terms

What was effect of the purges? First we must remember that Stalin's distrust and purges of Red Army started with the Tukhachevsky affair, a supposed conspiracy in the highest echelons of the Red Army to replace him. It then went down through the ranks, in typical Soviet style brutality and stupidity, nevertheless the gist of that affair remained - Red Army troops in or near major centers like Moscow, Kiev or Leningrad could be used for the coup. As such, Red Army and Navy commands far from the capital (like for example the Soviet Far East) were somewhat shielded from the purges as they were not so interesting to Stalin. Of course, the local NKVD still had to fulfill its quota of arrested (and executed) "traitors", but they were reluctant to move against higher Red Army officers without the nod from Moscow. Note that this could change quickly - both Grigory Shtern and Yakov Smushkevich were arrested and executed when they went were moved from their Far East commands on the orders of Stalin himself. But overall, officers in the Far Eastern Military District were much more secure in their posts (and had greater level of autonomy) than those in the Leningrad Military District which was responsible for the war against Finland.

I suggest you to look at the wikipedia articles on both wars. They are pretty exhaustive and you will be able to catch the differences that led two such different outcomes.

Capabilities of Red Army in 1939

In 1939, the Red Army had plenty of light and medium tanks alongside plethoric infantry and artillery made of towed pieces. Its air power was lagging behind in terms of technical and tactical capabilities, but it could perform good attacks against ground targets.

Red Army had the use of the experience of the Spanish War, where it had faced the Nationalists and the Italian Army. Land Italian armies were quite comparable to land Japanese armies.

Weaknesses of Red Army in 1939

The weaknesses however, were numerous:

• Lacks in command and logistics
• Lacks of training in interarm warfare
• Poor infantry: very static, unable to maneuver efficiently

Khalkin Gol battle

The battle of Khalkin Gol, against the Japanese, allowed the Red Army to use massively its advantages. Ground was flat or a little hilly, with grass and few trees. Soviet firepower could largely be used against the Japanese forces, that were inferior in that domain: this led the Japanese to fail in attacking and breaking through Soviet forces.

Then there was an operational pause, during which Jukov organized the counter-offensive: he went over the logistic difficulties by taking time and harassing everyone that tried to counter him. This allowed the Red Army to push rationally on Japanese forces, despite fierce fight, with tanks and heavy artillery.

But note that despite being well equipped overall in mortars and machine guns, the Soviet infantry proved inferior to its Japanese counterparts. And in the air, Soviet air forces were initially beaten before having reinforcements that allowed them to keep the air unsecure for Japanese pilots.

Later, during Winter 1940, the situation was overly difficult in comparison:

• Winter: this worked against an unprepared Soviet army
• Red Army was on the offensive and not defensive
• Ground was full of trees and swamps. Like really full The Red army was tied to a few roads and could not express its long range firepower. The lack of training of the infantry fully expressed and led to disaggregation of the units
• In the air, Soviet aviation could not help tactically because of the weather and the difficulty to target, and tried inefficient strategic bombings

Limitations

You should note, still, that the Red Army ultimately managed to break Finnish resistance:

• On the center, the tactic of the mottis allowed the Soviets to defend against Finnish attacks and pin down their troops.
• On the South front, they eventually could organize good attacks with interarm cooperation that led to breakthroughs of Finnish blockhaus

On the other hand, against the Japanese, Red Army lost tanks because of unorganized actions, especially during counter-attacks organized too fast, before Jukov operational pause.