Sullivan was the third son of a schoolmaster in Somersworth, New Hampshire, on the 18th of February 1740. He read law with Samuel Livermore of Portsmouth, New Hampshire, and began its practice in 1764 when he moved to Durham. He annoyed many neighbors in his early career, when he was the only lawyer in town, with numerous suits over foreclosures. But by 1772, he was firmly established and began work to improve his relations with the community. In 1773 Alexander Scammel joined John Sullivan's law practice.
He was sent by Durham to the colony's general assembly, and built a friendship with the royal governor John Wentworth. As the American Revolution grew nearer, he began to side more with the radicals. In 1774 the first Provincial (or rebel) Congress sent him as a delegate to the Continental Congress.
In 1772, he had been commissioned a major of New Hampshire militia, and on December 15, 1774, he and John Langdon led an expedition which captured Fort William and Mary at New Castle. In 1775 he was returned to the Congress, but when they appointed him a brigadier general in June and a major-general in August 1776, he left to join the army at the siege of Boston. He commanded a brigade in the siege of Boston.
After the British evacuated Boston in the spring of 1776, Washington sent General Sullivan north to replace the fallen John Thomas as commander in Canada. He took command of the sick and faltering invasion force, led an unsuccessful counterattack against the British at Trois-Rivières, and withdrew the survivors to Crown Point. This led to the first of several controversies between Congress and General Sullivan, as they sought a scapegoat for the failed invasion of Canada. He was exonerated and promoted to major general on August 9, 1776.
Sullivan rejoined Washington and was placed in command of the troops on Long Island to defend against British General Howe's forces about to envelop New York City. But then, on August 23, Washington split the command between Sullivan and General Israel Putnam. Confusion about the distribution of command contributed to the American defeat at the Battle of Long Island four days later. Sullivan's personal bravery was unquestioned, as he engaged the Hessian attackers with a pistol in each hand, however he was captured.
As a prisoner under parole, he carried letters from Admiral Richard Howe to the Congress. When the resulting peace discussions on Staten Island fell apart in September 1776 some in the Congress, particularly John Adams, found fault with Sullivan.
General Sullivan was released in a prisoner exchange in time to rejoin Washington before the Battle of Trenton. There his division secured the important bridge over the Assunpink Creek to the north of the town. This prevented escape and ensured the high number of Hessian prisoners captured. This route is now the main road in Ewing Township, New Jersey called "Sullivans Way". In January 1777. Sullivan also performed well in the Battle of Princeton.
In August, he led a failed attempt to retake Staten Island. Again Congress found fault, but he was exonerated by the court of inquiry. This was followed by American losses at Brandywine and Germantown. Congress was frustrated by the continued British occupation of Philadelphia, but since Washington was the only man holding the army together, they made Sullivan the scapegoat.
In early 1778 he was transferred to the unimportant post of Rhode Island where he commanded the largely unsuccessful Battle of Rhode Island in August 1778.
In the summer of 1779, Sullivan led the Sullivan Expedition, a massive campaign against the Iroquois in western New York. During this campaign, troops destroyed a very large Cayuga settlement, called Coreorgonel, on what is now the southwest side of Ithaca, New York.
He pushed his troops so hard that their horses became unusable, and killed them on this campaign, creating the namesake for Horseheads, New York. The lukewarm response of the Congress was more than he could accept. Broke, tired, and again opposed by Congress, he retired from the Army in 1779 and returned to New Hampshire.
In December, he was exchanged, succeeded Maj. Gen. Charles Lee in command of the right wing of Washington's army, in the battle of Trenton led an attack on the Hessians, and led a night attack against British and Loyalists on Staten Island, on August 22, 1777. In the battle of Brandywine on September 11, 1777, he again commanded the American right. He took part in the Battle of Germantown on October 4, 1777. In March 1778, he was placed in command in Rhode Island, and in the following summer plans were made for his cooperation with the French fleet under Count d'Estaing in an attack on Newport, which came to nothing. Sullivan after a brief engagement on August 29 at Quaker Hill, at the north end of the island of Rhode Island, was obliged to retreat.
In 1779, Sullivan, with about 4,000 men, defeated the Iroquois and their Loyalist allies at Newtown, New York, on August 29, burned their villages, and destroyed their orchards and crops. Although severely criticised for his conduct of the expedition, he received, in October 1779, the thanks of Congress. In November, he resigned from the army. Sullivan was again a delegate to the Continental Congress in 1780-81 and, having accepted a loan from the French minister, Chevalier de la Luzerne, he was charged with being influenced by the French in voting not to make the right to the north-east fisheries a condition of peace.
At home Sullivan was a hero. New Hampshire returned him as a delegate to the Continental Congress in 1780. But he still had opponents there. In 1781 when he borrowed money from the French minister to Congress, they accused him of being a foreign agent.  He resigned from the Congress in August 1781.
Back home again, he was named the state's attorney general in 1782 and served until 1786. During this same time he was elected to the state assembly, and served as speaker of the house. He led the drive in New Hampshire that led to ratification of the United States Constitution on June 21, 1788. He was elected President of New Hampshire (now Governor) in 1786, 1787, and in 1789.
When the new federal government was created, Washington named him federal judge for the District Court in New Hampshire in 1789. While his health prevented his sitting on the bench after 1792, he held the post until he died on January 23, 1795, aged 54, at his home in Durham. He was interred in the family cemetery there.