Were the ancient Israelites a caste society?

This is going to sound simplistic and possibly self-evident but Israel was a tribal society. And especially in light of Israel's ancient Hebrew texts this wasn't just a general 'tribal' notion but was delineated specifically historically, and by name, according to a fixed number of tribes. There is some debate whether there were 12 or 13 tribes since Ephraim and Manasseh, as Joseph's descendants, double Joseph's tribal representation (Joseph being one of Jacob's [Israel's] 12 sons). So there were in reality 13 tribes but only 12 were allotted land and the Levites in contrast were assigned specific cities in each of the 12 tribal lands of Israel and were subject to the hospitality of the host tribe.

The Levites indeed had a special status, but were the only tribe singled out in such a way for special roles in relation to temple worship, handling of agrarian tithes, and having no land inheritance, among other things.

The reason I go into this is because the Torah (Pentateuch or Law of Moses) spends far more time on inter-tribal relations than anything that could be considered 'caste' assignments in some heirarchical manner. For example the book of Numbers takes under consideration what happens when a woman in one tribe marries a man from another tribe in terms of inheritance transfer - out of a concern that one tribe may become so small from inheritance transfers over time that it would one day cease to exist. There was even a story (found in Numbers 27) of a man named Zelophahad who only had daughters, and the matter of the daughters' inheritance was brought before Moses when their father died since the rules at the time were not clear.

Even with the Levites though, they were seen neither as superior nor inferior to the other tribes in terms of the whole, and all tribes were expected to treat their brothers and sisters fairly and with equity; although of course the High Priest would have been one from the Levite ranks, and specifically had to be a descendant of Aaron, who was an individual of note (but then again many of the other tribes also supplied individuals of note in the form of judges, prophets, and kings). But hereditary succession of rulers and priests is a common feature of almost all ancient cultures and nations and can't be used exclusively to make a 'caste' argument.

Another answerer brought up slaves, but in that regard Israel's practice there must be compared to that of the entire Ancient Near East since all nations back then practiced it, especially enslavement of POWs and putting them to menial labor. However, that is neither prescribed by the Hebrew Biblical texts as something that should be done, nor is it discussed as a predetermined "lot in life" for a certain class of people as if it were a manifestation of the ideology of the Israelite religion in obedience to some kind of teaching or commands to enforce slavery.

I don't think Israel can be said to be based on a caste system unless one is willing to say that the majority of ANE nations were also based on a caste system (i.e. Egypt, Babylon, Hittite Empire, Assyria, Aram, Phoenicia, etc.).

Postscript: It is, however, worth noting – by comparison – that the Greek philosopher Plato did come up with some ideas of how society ought to be structured which bears some strong resemblance to caste systems. He even believed that each child was born with a different kind of metal heart, ranked from more precious to less precious metals, with the kind of metal they were born with determining their role in life. See this article on Plato's Five Regimes for more on that.

Ancient Israel was not a "caste" society, at least not in the sense that we know it in India.

You appear to be referring to the fact that the priests were drawn from the Levites, one of 12 tribes of Israel. That appeared to have been a "convenience" thing, to formalize the fact that a disproportionate number of priests (and temple workers) came from that tribe.

But that was exceptional; no other occupation was reserved for one tribe, and no other tribe "dominated" one occupation. And the Levites were not considered "better" than anyone else; if anything, they may have been consigned to that status for the tribes "past sins". So there wasn't a caste hierarchy built upon such specialization.