The refusal by Ralph Northam, the Democratic governor of Virginia, to resign after the revelation of a racist photograph is threatening his party’s political fortunes in Virginia, where Democrats are on the brink of consolidating power after a decadelong rise in the once-conservative state.

With Northam’s turmoil erupting during a legislative session in an election year, Democrats and Republicans said Sunday that his fragile hold on power risked his party’s policy ambitions and its aspirations for this fall, when control of both the state’s legislative chambers is expected to be bitterly contested.

“You can’t govern without a mandate, and all you’re going to do is make things worse for the state,” said Rep. A. Donald McEachin, a Democrat who served alongside Northam in the Virginia Senate.

Northam met with some of his staff members Sunday night, prompting speculation that he might announce his resignation during the Super Bowl. Both chambers of the Legislature are scheduled to meet Monday morning for sessions that could bring fresh condemnations of the governor. As of Sunday evening, Lt. Gov. Justin Fairfax, who would succeed Northam if he resigned, had not been notified that the governor was stepping down.

Northam’s troubles began Friday with the surfacing of a photograph on his medical school yearbook page, which showed a person in blackface posing with another in a Ku Klux Klan robe. The governor at first acknowledged that he was one of the figures in the image, and then denied it Saturday, all while drawing widespread calls for his resignation.

Two years ago, Democrats picked up 15 seats in the House of Delegates, where they had been locked out of the majority for more than two decades. They are now two seats away from control in both chambers. The biggest prize in controlling the statehouse would be the power, under current law, to draw congressional and legislative districts after the 2020 census.

Even to his Democratic allies, Northam now seems hobbled.

“You’ve got to work as one unit to move your commonwealth forward, and he’s just not going to have that ability to do it,” Terry McAuliffe, a Democrat who preceded Northam as governor, said on CNN’s “State of the Union.”

This article originally appeared in The New York Times.