Was slavery really on the way out in the antebellum USA?

The book Slavery by Another Name (Douglas Blackmon) argues compellingly that, post-Reconstruction, laws were put in place that both allowed prisoners to be used as slave labor (which arguably continues to this day) and that new laws that made it extremely easy to incarcerate people (vagrancy laws being an example) and keep them incarcerated were enacted. This suggests that slavery would have continued without the Civil War -- the economic incentive to have cheap laborers who have no alternative (which arguably, again, exists in modern America -- not just prisoners but also illegal aliens and simply poor people who are unable to break out of the cycle of poverty) is a powerful one.

A different question but related Slavery economically advantageous?

If cotton is taken to be a significant driver of the practice of slavery it can be argued that slavery was not in decline leading up to the Civil War. Cotton exports summary:

• 1820 America Exported 250,000 bales worth $22 million
• 1840 America Exported 1.5 million bales worth $64 million
• 1860 America Exported 3.5 million bales worth $192 million

America produced 3/4 of the world's cotton up from 9% in 1801. The invention of the cotton gin and Walter Burling introducing a new cotton strain from Mexico that had larger bolls and easier to pick helped make cotton such an significant economic activity of America. Cotton depended on slaves and the continued growth of the industry required expansion of the practice of slavery. By 1860 4 million of 4.5 million African Americans were slaves and almost all of them were owned by Southern planters.

"Capitalism in America; An Economic History of the United States", Alan Greenspan and Adrian Wooldridge; pages 74 - 79

Yes, slavery was ending. That's why the South seceded to preserve slavery. The South had been through a nearly 90-year political struggle over protecting their legal right to keep slaves and they had lost. Secession was their last-ditch hail marry pass at trying to preserve it. If succession had been successful, then perhaps the abolitionist cause would have been rolled back for a time; however, even the Southern leadership was forced to address Emancipation as the war turned against them. (see Cleburne-Davis proposal Southern Emancipation.

if slavery was really fizzing out by itself, how to explain the Kansas–Nebraska Act and the Dredd Scott decision which are usually understood as the slavers' onslaught?

You put your finger on why slavery was doomed. The Kansas-Nebraska Act took all previous restrictions on the growth of slavery off the table. It also took all previous guarantees of slave states maintaining their political equilibrium with free states off the table. The Kansas Nebraska Act stated all new states would decide for themselves whether to be free or slave and not have this decision dictated too by the Federal Government as a condition of statehood. While that doesn't sound anti-slavery, that was the net effect. The problem for supporters of slavery were that new states would primarily be populated by the more populous northern states and immigrants from slave-free-Europe. Additionally, folks willing to relocate and endure the hardships of the frontier states were overwhelmingly more likely to be poor laborers. Poor laborers were less likely to be slave owners, and more likely to see themselves in competition with slaves for jobs. For these reasons cultural, and economic; given a choice new states would vote to be free as both Kansas (January 29, 1861) and Nebraska (March 1, 1867) did when they entered the Union. This effect of the Kansas Nebraska act became very apparent to the South. South Carolina, the first state to leave the Union in the prelude to the Civil War, did so about a month before Kansas entered the Union as a free state. The two events free Kansas entering the Union and pro Slavery South Carolina leaving, were directly related.

Kansas Nebraska Act meant that as the west opened up, the vast majority of new states would be free. The south which had enjoyed an equilibrium in the Senate that allowed it to block any legal challenge to slavery for the first 90 years of the union would lose that ability.

The Kansas Nebraska act(1854) had replaced the Missouri Compromise(1819) which legislatively protected the balance between slave and free states. Kansas Nebraska Act blew apart the South's political power, not in 1854, but the writing was on the wall and everybody knew it, and one of the important pieces which ultimately lead to the first wave of southern secession.

As for the Dred Scott decision (1857) again, why slavery was doomed. You are right, on its merits the Dred Scott decision would seem to support slavery. It effectively legalized slavery in the North. It allowed southerners to travel in the North with their slaves protected from local laws. It meant federal agencies could be employed in the free North to repatriate run-away slave even when doing so violated state laws. In some cases, it meant the kidnapping of free African Americans and impressment into slavery as all that was required to name a person an escaped slave was an affidavit.

By giving the South all this power over the north, and superseding state laws, the federal government made slavery a central political issue in the north. The north could no longer ignore it as something which was happening somewhere else. Now it was happening in their own states. This unified and focused the north to politically oppose slavery. That lead to the popularity and mainstreaming of the abolitionist movement in the North. It lead to the dissolution of the Whig Party and condemnation of it's 90 year track record of compromising on the slave issue. It lead to the rise of the new Republican Party, a party dedicated to the political destruction of slavery.

Dred_Scott vs Sandford
Although (US Supreme Court Chief Justice Roger B.)Taney believed that the (Dred Scott) decision represented a compromise that would settle the slavery question once and for all by transforming a contested political issue into a matter of settled law, it produced the opposite result. It strengthened Northern opposition to slavery, divided the Democratic Party on sectional lines, encouraged secessionist elements among Southern supporters of slavery to make bolder demands, and strengthened the Republican Party.

.

Dred Scott Decision
The decision inflamed regional tensions, which burned for another four years before exploding into the Civil War.

.

Reaction to the Dread Scott Decision
This loaded decision, which was supposed to solve the slavery question once and for all and more importantly mitigate the nation's growing sectional crisis, ended up creating more tension in the country between the North and South. The reaction to the decision varied by region and political party, with it being criticized by northerners and Republicans, and praised by southerners and Democrats. The nation's intense reaction to the Dred Scott decision not only had an effect on politics in the late 1850s, but would also serve as one of several precipitates for the ultimate breakdown in American politics, the southern secession and Civil War.

.

Dred Scott Decision Still Resonates Today
The decision also made the Republican Party a national force, and led to the division of the Democratic Party during the 1860 presidential elections.

The growing power of the Republicans, who received considerable support from the northern states, directly led to fears in the South that slavery would be ended, and those fears started the momentum for secession and the Civil War.

Dred Scott enraged and radicalized the North.

Granted the question wasn't to fight the civil war and end slavery in 1865, or to not fight the war and end slavery in 1866. Without the civil war slavery may have taken decades more to end. But without legislative protection in the Senate, it could not have long continued. Supreme Court Dred Scott decision was so outrageously out of step and offensive to the majority it set the stage for the confrontation. The decision escalated the collision course the two sides were on and removed much of the room for compromise which had existed in the north since revolutionary war days. The South could not hope to preserve slavery. Dred Scott made the institution the pre-eminent issue, and the Kansas Nebraska act had removed it's most reliable protection. There was now no neutral ground which could support compromise and the most popular political party in the north now had the elimination of slavery as one of their core founding principles.

The South was check mated, and they knew it. That is why the whig party which had participated in the great compromises which had allowed slavery to prosper became defunct, and was replaced by the abolitionist Republican Party with the mandate to end slavery. That is why the South they refused to participate in the 1860 election. That is why the North elected an abolitionist President even knowing could mean war. That is ultimately why the South left the Union. Because the Union was no longer a place which would support the institution which their society and economy was organized around, slavery.

Here is another interesting complement to the excellent answers I have already received. In a recent article in Aeon magazine Matthew Karp unearthes the following very pertinent information:

And for antebellum Southern writers, the destiny of the US was manifestly both imperial and slaveholding. Their future was a future where slavery would continue to thrive. The black slave population would reach 10.6 million in the year 1910, according to the calculations of the New Orleans editor (and later US Census superintendent) James D B De Bow. Later, an Alabama politician quoted another estimate that put 31 million American blacks in chains by 1920. The Richmond-based Southern Literary Messenger, in an 1856 article exploring ‘the condition of the slavery question in the year 1950’, offered the most grandiose prediction of all, that the US slave population would ‘amount to 100,000,000 within the next century’.

His article is aptly called "In the 1850s, the future of American slavery seemed bright".

No, not even close.

Alan T Nolan lists this as one of the components of the Lost Cause Myth in his essay "The Anatomy of the Myth", collected in the book The Myth of the Lost Cause and Civil War History (ed by Gary Gallagher and Nolan, Indiana University Press, 2000). McPherson says in Battle Cry that slavery was more firmly entrenched in 1860 than it had been in 1820. By 1860 the "stock" of enslaved persons was the single most valuable concentration of wealth in the US – more valuable than all the railroads and manufacturing concerns in the country PUT TOGETHER. The Mississippi History Now website has a page that says:

[C]otton was America’s leading export, and raw cotton was essential for the economy of Europe. The cotton industry was one of the world’s largest industries... In many respects, cotton’s financial and political influence in the 19th century can be compared to that of the oil industry in the early 21st century.

Groups don't typically just give up that much wealth/power, without an objection.

By 1860 it was illegal in the South for antislavery literature (including many Northern newspapers) to be sent in the mail: Southern inspectors were opening mail. (See for example here and here and here and here.) Free speech was infringed upon: it was illegal to voice antislavery opinions. By 1860 it was illegal in many Southern states to even BE a freed Negro: if you were an enslaved person and somehow were given your freedom, you'd better get the hell out of the state FAST. Kenneth Stampp writes in The Peculiar Institution (1956) that by 1860, many Southern courts had started reviewing wills where a dying slaveowner freed his enslaved persons, and overruling the will, ordering that the enslaved persons be kept in slavery. It seems that in the South, a slaveowner could do anything with his enslaved persons except free them. So the laws in the slave states were tighter by 1860 than they had ever been.

The Federal situation was even worse for enslaved persons (and freed blacks). The Fugitive Slave Law of 1850 was more draconian than anything in effect back in 1820: it was quasi-legal kidnapping. The 1857 Dredd Scott decision purported to make Negros ineligible for citizenship in any state – INCLUDING in the 5 states where blacks had held citizenship and been able to vote during the Constitional ratification era. In other words, the 1857 Supreme Court ruling took away black citizenship in 5 states where they had held citizenship since 1789. There was a very real fear in the North that the next ruling from Roger Taney's court could make it unconstitutional to EXCLUDE slavery from any state. Lincoln stressed this in his "House Divided" speech at Springfield in 1858, quoted on the Dred Scott wikipedia page. That same page contains a reference to Southern radicals boasting that the coming decade (after the Dred Scott decision) would see slave auctions on Boston Common. (Lemmon v. New York might have supplied the Court with an opportunity.)

And of course the slave states were looking to expand slavery into Mexico and Cuba. Many Southerners could foresee a paradise of slavery extending all around the Carribean basin. See here and here.

This census.gov link shows the actual number of enslaved persons growing from just under 700,000 in 1790, to just under 4 million in 1860:

That does not look like an institution that is dying out. It's an average growth rate of 2.5% per year, 28% per decade. The ending of the slave trade is not even a blip on this growth chart: 293,000 more slaves in 1810 over 1800, and 347,000 more enslaved persons a decade later. (And, of course the ending of the slave trade would not make a dent in this growth, once it was established. A few shiploads, or few dozen, is nothing compared to how many children 893,000 people can have.) The "delta" grew every decade, as you would expect with exponential growth. In 1860 there were 750,000 more enslaved persons than there had been in 1850, the highest delta ever. At that rate, there would have been over 5 million enslaved persons in 1870, an increase of 1.1 million over 1860.

Gail Jarvis and the Dunning school present an outrageously deceptive misinterpretation of some facts. First:

"Plantation owners and their families were well aware of the slave revolts in Haiti and other Caribbean island where hundreds of Whites had been slaughtered. These stories, coupled with reports of slave uprisings in the American South, certainly must have conciliated many a hardened pro-slavery stance."

Yes, Southern slaveowners were well aware of the revolts in Haiti and the Carribean, and reports of uprisings in the south. Stampp documents that in fact they were hyper-aware of them. That awareness created at atmosphere of suspicion and suppression. This did the exact opposite from "conciliating" hardened slavers. Instead, those reports induced periodic panics and reprisals, where slave patrols tortured and murdered enslaved persons who were suspected of plotting. This was an integral part of the regular pattern of terror and oppression that kept enslaved persons in line. The exact opposite of conciliation.

Southern awareness of Haiti et al extended to the decision to secede. Southerners told each other that Black Republicans would turn the South into another Haiti. (See Apostles of Disunion by Charles Dew, 2001) They said they had to secede to protect their families and especially their daughters.

Second:

"there were more than 250,000 Free Persons of Color in the South, not only in major cities like Charleston and New Orleans but in smaller towns."

250k free blacks in the South is a drop in the bucket compared with the 4 million enslaved persons: just 6% It is only one-third of the number of NEW enslaved persons in 1860 over 1850. The argument is absurd, maintaining that the 6% is "representative" while ignoring the other 94%. It is a deliberate distortion.

Stampp also writes about the profitability of slavery. He says that slaveowners typically claimed that they lost money on enslaved children whom they fed and clothed thru childhood, and then later sold. Those slaveowners would deny that they were in the business of "farming slaves", which had negative connotations. They fed those enslaved children out of the goodness of their hearts, and lost money on the deal. But Stampp reviews the sale prices for those enslaved persons, as recorded in various contemporary publications, and concludes that raising & selling enslaved persons was big money for those plantation owners large enough to do it. He does not go into specific numbers; but this directly contradicts Ulrich Phillips' assertion. The profitability of plantation slavery was not tied only to the production of cotton, but to "farming slaves" also.

No, the idea that slavery was somehow on its way out in the antebellum USA is a complete fabrication by the Dunning school, without a shred of evidence supporting it. It's so at variance with the demonstrable facts that it has to be an outright lie, an attempt to deceive. And in fact, that's what the whole Dunning School of history was: propaganda to support & justify the Jim Crow system. Here is Eric Foner on it 10:

The traditional or Dunning School of Reconstruction was not just an interpretation of history. It was part of the edifice of the Jim Crow System. It was an explanation for and justification of taking the right to vote away from black people on the grounds that they completely abused it during Reconstruction. It was a justification for the white South resisting outside efforts in changing race relation because of the worry of having another Reconstruction.

All of the alleged horrors of Reconstruction helped to freeze the minds of the white South in resistance to any change whatsoever. And it was only after the Civil Rights revolution swept away the racist underpinnings of that old view — i.e., that black people are incapable of taking part in American democracy — that you could get a new view of Reconstruction widely accepted. For a long time it was an intellectual straitjacket for much of the white South, and historians have a lot to answer for in helping to propagate a racist system in this country.

It is interesting to speculate on when slavery would have ended, absent the civil war. Two possible "outer boundaries" for when plantation slavery would have ceased being profitable are the boll weevil infestation in the early 1920s, and the invention of the first really successful mechanical cotton picker or combine, the International Harvester model "H-10-H", in 1942. It's difficult to imagine slavery in America in the 20th century: those suggestions are just pure speculation about the absolute latest possible dates when plantation slavery might still have been economically viable. But of course enslaved persons could work in factories and mines, too.

Lincoln and the Republicans wanted to put slavery "in course of ultimate extinction", by limiting it to the existing slave states and keeping it out of the territories and any new states. The idea was that if slavery was kept where it was, then it would eventually die out, maybe with the aid of "compensated emancipation" where the government would pay the slavowner some discounted value for the freed slaves. I read somewhere that Lincoln's own estimate for how long it would take for that to happen was around 50 years. (I will update if I ever find my source for that.)

This of course is exactly what the slave states were afraid of, and why they seceded. Ironically, secession started the ball rolling that did in fact end slavery. But before secession slavery was still going very strong, with no end in sight.