_{This post is intended primarily as a supplement to 0range's answer; the reader should also be aware that Hinduism might not necessarily have a monolithic view on the matter.}

The famous law code, Manusmriti provides for punishment to h*m*sexual men and women.

Manusmriti says that if a girl has sex with another girl, she is liable for a fine of two hundred coins and ten whiplashes.

But if lesbian sex is performed by a mature woman on a girl, her head should be shaved or two of her fingers cut off as punishment. The woman should also be made to ride on a donkey.

In the case of h*m*sexual males, Manusmriti says that sexual union between two men brings loss of caste. If a man has sex with non-human females or with another man or indulges in anal or oral sex with women he is liable for punishment as per the Painful Heating Vow.

Arthashastra of Kautilya — a treatise on politics — also mentions h*m*sexuality. But the book makes it a duty of the king to punish those indulging in h*m*sexuality and expects the ruler to fight against the social evil.

— India Today: 10 Instances of Homosexuality Among LGBTs in Ancient India.

However, in another Hindu text, the Manusmriti, there are various punishments for h*m*sexuality.

Girls who had sex with other girls were punished with two hundred coins and ten whiplashes.

A mature woman having sex with a girl was punished by having her head shaved or two of her fingers cut off, and she was also made to ride on a donkey.

In the case of h*m*sexual males, the Manusmriti dictated that sexual union between two men brought the loss of caste.

In the Arthashastra, a 2nd century BCE Indian treatise on statecraft, h*m*sexual intercourse was treated as an offence, although several kinds of heterosexual intercourse were punished more severely.

— Wikipedia: LGBT rights in India.

The Arthashastra, a 2nd century BCE Indian treatise on statecraft, mentions a wide variety of sexual practices which, whether performed with a man or a woman, were sought to be punished with the lowest grade of fine. While h*m*sexual intercourse was not sanctioned, it was treated as a very minor offence, and several kinds of heterosexual intercourse were punished more severely.

Sex between non-virgin women incurred a small fine, while h*m*sexual intercourse between men could be made up for merely with a bath with one's clothes on, and a penance of eating the five products of the cow and keeping a one-night fast – the penance being a replacement of the traditional concept of h*m*sexual intercourse resulting in a loss of caste.

— Wikipedia: LGBT history in India & Homosexuality in India: History.

Homosexuality and pederasty was rare in medieval Hindu society with Al-Biruni saying that Hindus greatly disapproved of it.

— Wikipedia: LGBT history in India.

Devdutt Pattanaik summarizes the place of h*m*sexuality in Hindu literature as follows: though not part of the mainstream, its existence was acknowledged but not approved.

— Wikipedia: Homosexuality in India: History & LGBT rights in India.

The Dharmsastras, especially the later ones, like the Vashistha Dharmasutra, prescribed against non-v***nal sex. The Yājñavalkya Smṛti prescribes fines for such acts, including those with other men.

— Wikipedia: Hinduism and LGBT topics & LGBT culture in India.

The Arthashastra of Kautilya represents the principle text of secular law and illustrates the attitude of the judiciary towards sexual matters. Heterosexual v***nal sex is proposed as the norm by this text and legal issues arising from deviation therefrom are punishable by fines and in extreme cases by capital punishment. Homosexual acts are cited as a small offence punishable by a fine. It punishes non-v***nal sex with a small fine (4; 23; 326), however, women are fined less than men.

The Manusmriti's punishments are light. [...] According to XI.68, a man who engages in such acts is traditionally considered to lose his caste, though Ruth Vanita suggests the prescriptions by Manusmriti act as a substitute.

Verses 8.369-370 of Manusmriti, which prescribe punishment for a female having intercourse with a maiden, are wrongly thought to be [primarily] against same-sex activity between females by some modern authors like Wendy Doniger. However, verse 8.367 contains a similar punishment for all those who do it regardless of gender. The emphasis, Vanita states here, is on a maiden's sexual purity.

— Wikipedia: Hinduism and LGBT topics.

Ayoni, or non-v***nal sex of all types, are punishable in the Arthashastra. Homosexual acts are however treated as a smaller offence punishable by a fine while unlawful heterosexual sex have much harsher punishment.

Manusmriti prescribes light punishments for such acts. Vanita states that the verse about punishment for sex between a female and a maiden is [primarily] due to its strong emphasis on a maiden's sexual purity.

— Wikipedia: LGBT culture in India.

tl;dr

No, it would have been the Portugese, see here and here.

Questions

Was this the first time h*m*sexuality was outlawed in India? If not, who was the very first to do so in the Indo-Pak subcontinent? Was it the one of the Muslim rulers?

Answer: Not the British, the Portuguese (Portuguese India). During the Goa Inquisition commencing in 1560, more than 300 years before 1861 (i.e. Section 377 of the Indian Penal Code). Not a Muslim ruler, but the Catholics - in particular Portuguese inquisitors, the first one being Aleixo Díaz Falcão.

The earliest case of a conviction for "sodomy" I could find where the convicted is known by name is from 1589: Antonio de Matos, born in Hormuz, was sentenced to the galleys (i.e. being a galley slave) for life.

Sources: Alisa Meyuhas Ginio (1999): The Inquisition and the New Christians: The Case of the Portuguese Inquisition of Goa in The Medieval History Journal Vol 2, Iss 1, Stephanie Hassell (2015) Inquisition Records from Goa as Sources for the Study of Slavery in the Eastern Domains of the Portuguese Empire in History in Africa, Vol 42, pp.397-418) (scholarly papers behind paywalls, sorry); alternatively the wikipedia articles on the Goa inquisition and in the LGBT history of India.

Detailed answer

Let me detail the various aspects of this in the following order:

1. What exactly is being outlawed (h*m*sexual behavior, not h*m*sexuality)
2. Who would be expected to outlaw h*m*sexual behavior
3. Who was the first to outlaw h*m*sexual behavior anywhere on the Indian subcontinent
4. Who was the first to outlaw h*m*sexual behavior in all of the Indian subcontinent

1. What exactly is being outlawed

Note that it does not make sense to outlaw sexual orientations in the same way that it does not make sense to outlaw introversion, optimism, or perfectionism. Those behavioral patterns are parts of the personality that people cannot possibly do anything about. They are quite possibly already fixed at birth and most certainly partly hereditary.

Those who outlaw h*m*sexual behavior typically deny that h*m*sexuality exists, for ridiculous examples, see here and here. They see h*m*sexuals as sinners against some moral or behavioral code and outlaw the sin ("sodomy" etc.). That this leads to cultural animosity, harassment persecution of LGBT people as a group, and witch hunt like incidents, is a different question.

2. Who would be expected to outlaw h*m*sexual behavior

This is typical for religious fanatics, especially those in the tradition of the Abrahamic religions, although the degree to which h*m*phobia is seen as a central element of the religion varies quite a lot with time and across religions. Currently, h*m*phobia and persecution of h*m*sexuals is a major element of Wahhabi ideology and of Islamist fundamentalism (since the Wahhabis are sponsoring and funding it, to some degree together with Iran) which finds some appeal in conservative Muslim culture in general. This has been very different in pre-modern times; Muslim states tended to be much more tolerant and liberal. Christianity on the other hand has experienced the opposite development and is much more liberal today than it used to be in the late Middle ages and in early modern times (although certain right wing elements in many predominantly Christian countries would very much like to reverse this and return to Middle age zealotry).

3. Who was the first to outlaw h*m*sexual behavior anywhere on the Indian subcontinent

The British came relatively late in the colonization of India, in 1612, after the Portugese and the Dutch were already there and shortly before the French and the Danish arrived.

The Portugese were the first Europeans to arrive in India in 1498 and seize territories there as colonies from 1505. They lost no time to implement the Portugese inquisition against what they saw as "sodomites" and a lot of others.

4. Who was the first to outlaw h*m*sexual behavior in all of the Indian subcontinent

The British were the first and only Europeans that did conquer larger territories in the subcontinent. This did not occur before the 19th century, however, after they defeated the French. Note that the subcontinent was rarely a unified political structure until then, except for the periods of the Maurya empire the Mughal empire and possibly the Gupta empire (each at the time of their largest extent). Consequently, if the original question is referring to laws that would have been valid for all of the subcontinent, the answer is that there, was for almost all times in history, no one with the authority to create such a law.

While they only reorganized Indian criminal law after conquering the subcontinent, they would not have tolerated "sodomy" even before that in the areas under their control. Neither would the other European powers, though none of them would have been quite as brutal as the Portuguese inquisition.

Edit (Feb 2018): Since it was criticized that the answer might be missing the point, I tried to structure it better with a tl;dr section and sections detailing the various aspects. The contents and explanations are the same as in the first version of the answer.

Edit (Aug 2018): @J Asia: Thanks for helping improve this answer. I believe you were thinking about explicit evidence of persecution of lesbian sexual practices ("lesbian sodomy") which is discussed by the book you cite (Soyer, F., Ambiguous Gender in Early Modern Spain and Portugal (Brill, 2011), p.45.) and corresponds better to the date (1744) and names (Joao Cosme da Cunha) you give. However, the Goa inquisition commenced about 200 years earlier, in 1560 with inquisitor Aleixo Díaz Falcão by royal order of king João III following a request St. Francis Xavier made in 1546. Please see the linked wikipedia article on the Goa inquisition and the sources cited there. It was standard practice for the inquisition (especially in its Iberian tradition) to persecute h*m*sexual behavior, male "sodomy" in particular. But there is no reason to believe that they were much more lenient with "female sodomy" where it came to their attention. In fact, as Ginio (1999) details, sodomy accusations were quite common in the Goa inquisition, including false accusations brough by bribed witnesses. Ginio's evidence includes a tretise written in 1687 by Charles Dellon who had been subject to such an accusation himself.