What was unreasonable about Austria's ultimatum to Serbia?

Short answer: points 4-6 were unreasonable to the point of being unacceptable, because they gave Vienna so much power over Serbia that it amounted to a forfeiture of Serbian independence.

This isn't unique to the time period or Serbia. Countries generally are not happy to subjugate themselves to a hated enemy. Whether they could afford to resist is another matter, of course; but whether they bow to reality or not doesn't diminish the humiliation of harsh terms.

Lastly, Serbia indeed tried to appease Vienna. It (on paper, anyway, and conditionally) accepted every demand except one.

Much of the ultimatum could be deemed unreasonable. But with regards to being unacceptable, Serbia actually folded to every Austrian demand and accepted all terms except one:

6. to institute a judicial inquiry against every participant in the conspiracy of the twenty-eighth of June who may be found in Serbian territory; the organs of the Imperial and Royal Government delegated for this purpose will take part in the proceedings held for this purpose;

Bringing murderers to justice was probably not very controversial. The real issue is the second phrase, which is a thinly veiled demand for Serbia to submit to the jurisdiction of Austrian police. This was of course unacceptable, since it is tantamount to a surrenders of Serbian sovereignty without a fight. As the Serbian reply said:

The Royal Government considers it its duty as a matter of course to begin an investigation against all those persons who have participated in the outrage of June 28th and who are in its territory. As far as the cooperation in this investigation of specially delegated officials of the I. and R. Government is concerned, this cannot be accepted, as this is a violation of the constitution and of criminal procedure. Yet in some cases the result of the investigation might be communicated to the Austro-Hungarian officials.

A less righteous read of the refusal is that the Serbian government feared Austria would discover that Serbia knew about the assassination. Alternatively, Serbian leaders might (perhaps reasonably) simply fear that given the authority, the Austrians would simply frame them.

Two other points were also generally deemed unacceptable but "accepted" by Serbia:

4. to remove from the military and administrative service in general all officers and officials who have been guilty of carrying on the propaganda against Austria-Hungary, whose names the Imperial and Royal Government reserves the right to make known to the Royal Government when communicating the material evidence now in its possession;

5. to agree to the cooperation in Serbia of the organs of the Imperial and Royal Government in the suppression of the subversive movement directed against the integrity of the Monarchy;

These were ultimately was agreed to in principle by the Serbian government, but carefully laced with enough conditions so that non-compliance was all but assured. As was the case with point six, these articles were unacceptable because they would have given Austria extreme authority over Serbian internal affairs. Consequently, although it is usually said that Serbia accepted these terms, the Serbian reply actually noted that:

The Royal Government is also ready to dismiss those officers and officials from the military and civil services in regard to whom it has been proved by judicial investigation that they have been guilty of actions against the territorial integrity of the Monarchy; it expects that the I. and R. Government communicate to it for the purpose of starting the investigation the names of these officers and officials, and the facts with which they have been charged.

Which limited the arbitrary power to dismiss anyone they want that Austria sought to those proven "guilty", and:

The Royal Government confesses that it is not clear about the sense and the scope of that demand of the I. and R. Government which concerns the obligation on the part of the Royal Serbian Government to permit the cooperation of officials of the I. and R. Government on Serbian territory, but it declares that it is willing to accept every cooperation which does not run counter to international law and criminal law, as well as to the friendly and neighbourly relations.

Which placed restrictions of "international law and criminal law" on what the Austrian government could do under the excuse of pursuing terrorists.