What is the biblical basis for NOT keeping a literal physical Sabbath rest every seventh day?

The need for a literal, physical Sabbath rest began at the Fall in the Garden of Eden when God cursed the ground for the sake of Mankind:

“Because you have listened to the voice of your wife and have eaten of the tree of which I commanded you, ‘You shall not eat of it,’ cursed is the ground because of you; in pain you shall eat of it all the days of your life; thorns and thistles it shall bring forth for you; and you shall eat the plants of the field. By the sweat of your face you shall eat bread, till you return to the ground, for out of it you were taken; for you are dust, and to dust you shall return.” - Genesis 3:17-19

Prior to that event Adam enjoyed the provision of the produce of all of the trees in the Garden save one:

The LORD God took the man and put him in the garden of Eden to work it and keep it.  And the LORD God commanded the man, saying, “You may surely eat of every tree of the garden, but of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die.” - Genesis 2:15-17

Adam was created to be a servant and, as the above verse shows, he had work and responsibilities in the garden of God but this work was not toil and vanity, for those (and the need for physical rest) entered as a result of the curse and expulsion from the Garden.

Since God rested from all of His creative labors on the seventh day, by which we are to understand a lasting rest, not as though He commenced work again on day eight, we can envision Adam working in the Garden during God's Sabbath rest.

Jesus Christ, as the second Adam, also went about doing the works of God, not only on the weekly Sabbath prescribed by the Law (many examples too numerous to list) but also always during that rest of God's perpetual seventh day.

When Adam was expelled from the Garden he was essentially expelled from God's presence and rest. Ruined by sin and toiling in a cursed creation apart from the presence of God, Mankind needs not only a prescription for physical rest but a prescription for remembrance of the Creator and Sustainer of all despite the curse. Herein lies the reason that God's people, when placed under the tutelage of the Law to bring them to Christ and, through Christ, back to Himself were given a weekly rest Day. The weekly Sabbath, like all of the Law, is intended to bring us to Christ:

Why then the law? It was added because of transgressions, until the offspring should come to whom the promise had been made, and it was put in place through angels by an intermediary. Now an intermediary implies more than one, but God is one. Is the law then contrary to the promises of God? Certainly not! For if a law had been given that could give life, then righteousness would indeed be by the law. But the Scripture imprisoned everything under sin, so that the promise by faith in Jesus Christ might be given to those who believe. Now before faith came, we were held captive under the law, imprisoned until the coming faith would be revealed. So then, the law was our guardian until Christ came, in order that we might be justified by faith. But now that faith has come, we are no longer under a guardian, for in Christ Jesus you are all sons of God, through faith. - Galatians 3:19-26

Christ has declared Himself to be Lord of the Sabbath and this declaration is given in explanation of why His disciples do what is unlawful on the (weekly) Sabbath:

But when the Pharisees saw it, they said to him, “Look, your disciples are doing what is not lawful to do on the Sabbath.” He said to them, “Have you not read what David did when he was hungry, and those who were with him: how he entered the house of God and ate the bread of the Presence, which it was not lawful for him to eat nor for those who were with him, but only for the priests? Or have you not read in the Law how on the Sabbath the priests in the temple profane the Sabbath and are guiltless? I tell you, something greater than the temple is here. And if you had known what this means, ‘I desire mercy, and not sacrifice,’ you would not have condemned the guiltless. For the Son of Man is lord of the Sabbath.” - Matthew 12:2-8

In saying that He is greater than the temple (by which we can infer the Law and the Levitical system) it is to be understood that the Sabbath that He is Lord of is that Sabbath rest of God which is greater than the weekly Sabbath of the Law (this, incidentally, is strong Trinitarian evidence). If He is Lord of the greater rest then He is assuredly Lord of the lesser.

And he said to them, “The Sabbath was made for man, not man for the Sabbath. So the Son of Man is lord even of the Sabbath.” - Mark 2:27-28

In saying that the (weekly) Sabbath was made for man, not man for the (weekly) Sabbath he demonstrates that weekly Sabbath keeping was not the original purpose of Mankind but rather to live within the perpetual Sabbath rest of God.

In this way the writer of Hebrews indicates that there remains a Sabbath rest for the people of God (under the Law) to enter:

Therefore, while the promise of entering his rest still stands, let us fear lest any of you should seem to have failed to reach it. For good news came to us just as to them, but the message they heard did not benefit them, because they were not united by faith with those who listened. For we who have believed enter that rest, as he has said, “As I swore in my wrath, ‘They shall not enter my rest,’” although his works were finished from the foundation of the world. For he has somewhere spoken of the seventh day in this way: “And God rested on the seventh day from all his works.” And again in this passage he said, “They shall not enter my rest.” Since therefore it remains for some to enter it, and those who formerly received the good news failed to enter because of disobedience, again he appoints a certain day, “Today,” saying through David so long afterward, in the words already quoted, “Today, if you hear his voice, do not harden your hearts.” For if Joshua had given them rest, God would not have spoken of another day later on. So then, there remains a Sabbath rest for the people of God, for whoever has entered God's rest has also rested from his works as God did from his. - Hebrews 4:1-10

Since

1. It is God's Sabbath rest that remains for God's people to (re)enter
2. That rest is God's intended purpose for Man
3. That rest is greater than the weekly Sabbath rest of the Law
4. Jesus is Lord of both and violates the lesser while honoring the greater
5. The greater rest represents restored fellowship with God

Those who are "in Christ" may demonstrate the greatest understanding of the Lordship of Jesus Christ by de-emphasizing the lesser and striving for the greater. Jesus has come to bring us into the pre-fall rest of God's perpetual Sabbath.

• This does not indicate that weekly Sabbath keeping is inherently wrong for Christians but that insistence upon it in order to "be" a Christian is, for then the lesser (Sabbath) is elevated above the greater (Lord of Sabbath):

As for the one who is weak in faith, welcome him, but not to quarrel over opinions.  One person believes he may eat anything, while the weak person eats only vegetables.  Let not the one who eats despise the one who abstains, and let not the one who abstains pass judgment on the one who eats, for God has welcomed him. Who are you to pass judgment on the servant of another? It is before his own master that he stands or falls. And he will be upheld, for the Lord is able to make him stand. One person esteems one day as better than another, while another esteems all days alike. Each one should be fully convinced in his own mind. The one who observes the day, observes it in honor of the Lord. The one who eats, eats in honor of the Lord, since he gives thanks to God, while the one who abstains, abstains in honor of the Lord and gives thanks to God. - Romans 14:1-6