Why didn't Japan expand into Ezo?

Japan is a country with a broad range of weather conditions and a lot of difficulties if you are an average rice farmer who just wants to survive. The northern part of Honshu (Touhoku) and Hokkaido is famous for harsh conditions for life even now. The traditional Japanese lifestyle is more about hot summers and mild winters, while the northern end of Honshu and Hokkaido has cold winter, a lot of snow (which is rare even in Tokyo metro area). If you go to Hokkaido, even now it is easy to recognize that buildings have heat insulation, central heating, etc unlike the rest of Japan.

The agriculture of Japan is based on rice, but Japan is one of the northernest places where rice is grown. While one can grow rice in Hokkaido, the settlement made much more sense after crops like potato and farm animals were introduced (during Meiji restoration with American help). That required extra effort, eg the largest and oldest university of the island, Hokkaido University in Sapporo was established as Sapporo Agricultural College by Clark. This kind of agriculture actually paid off: Hokkaido now an important agricultural center in spite of the low population.

In summary, the settlement of Hokkaido was far from simple, traditional Japanese agriculture, buildings, etc. didn't work very well there, therefore settlements would have not much payoff. Living and producing in Hokkaido became much more accessible introducing foreign know-how about living in colder climates and modern agriculture.

I think your presumption that Japanese power through military force was not employed in Ezo during Edo is wrong. You may read about Sakushin's revolt and it's suppression in 1669.

UPDATE: 9-18-2017 Here's a much better source Shakushain's War describing the rebellion and going into great depth about the background leading up to and following the rebellion.

Early in the Edo era (~1620) the Shogunate gave the Matsumae clan the right to build a castle on Ezo and the most importantly the right to monopolize all trade there - The Ainu could only trade with Matsumae representatives and it was the Matsumae representatives who decided the prices. This had a great economic impact on the Ainu right from the beginning of the Edo Era.

Interestingly however, the second reference persuasively describes a much more complicated set of circumstances leading to the rebellion: it started out as a feud between rival Ainu clans, which eventually spun out into a rebellion when the Japanese appeared to be favoring one side.

The revolt failed militarily as arrows and spears were no match for the muskets the Japanese were armed with. However, it was a blockade on any trade with the Ainu which really forced the remaining communities to abandon any attempts at resistance. They had long depended on rice obtained through trade with Japan ( since early Edo via Matsumae-clan monopoly only), and without the rice they were starving.

My own conjecture: Originally the Ainu never or rarely ate rice at all. Their population remained bound by the quantity of wildlife they could hunt and the wild nuts and vegetables they could gather. They lived in equilibrium with their environment. Once they started trading captured wildlife with the Japanese for rice, they began to voraciously deplete their natural resources while at the same time thriving on the rice they received in return. This in turn led to feuds between rival Ainu clans as increasing population vied for diminishing natural resources to sell for rice. (The second resource does describe both the competition for resources and the vanishing resources, e.g., bears being unavailable for traditional ritual).

Once this effective domination of the Ainu is put in perspective, one part of the OP's question still remains: Why didn't mass colonization of Hokkaido occur until the Meiji era?

A possible answer to that: Look what happened to Great Britain with their colonists in the thirteen colonies.

The people already living in the Ezo, the Ainu, were no joke militarily, and tangled regularly with the Mongolian Empire on Sakhalin, and later, the Ming in the Amur Valley. It's not clear the Japanese could have taken the island completely if they wanted to, not before they modernized in the 19th century.

This also highlights another point - the islands further north were more in reach of the mainland powers. Hokkaido, until the Meiji Restoration, was more useful as a frontier than a destination for colonization.