By JUSTIN POPE, AP Education Writer BOSTON - An Ivy League academic who served in President Clinton's cabinet, Harvard President Lawrence Summers is an unlikely conservative martyr. But after announcing his plans to resign, it looks like Summers is becoming just that. In his five years as Harvard president, Summers has supported ROTC on campus, suggested men may excel over women in the scientific elite partly because of genetics, and confronted a prominent professor, Cornel West, over the academic value of his rap CD. He's also argued that the school's brilliant minds should spend more time teaching, and should work more closely together to solve real-world problems. Conservatives, with few fellow-travelers running top universities, adopted Summers as one of their own. Now many of them say Summers' downfall underscores how those schools have lost touch with the country. "Larry Summers is a liberal, (but) he was trying to do the right thing," said David Horowitz, an outspoken critic of liberal faculty bias on campuses. "These universities have been taken over. It's 10 percent who got rid of him. They're hardline Stalinists. They're not liberals." Some moderates and even liberals hear at least some truth in what Horowitz says. "It's unfortunate that it's seen as an issue of liberal vs. conservative, because real liberals are horrified by the academic hard left," said Harvey Silverglate, a Boston civil rights lawyer and author of the book, "The Shadow University: The Betrayal of Liberty on America's Campuses." "Academic freedom can't survive the control by that cult," he said. Summers said Tuesday he would step down rather than continue to grapple with Harvard's core Faculty of Arts & Sciences, which passed a no-confidence vote in him last March and was poised to take another one this week. An economics professor, Summers said he would return to teach at Harvard after a year sabbatical. Students had backed Summers â the Harvard Crimson student newspaper lamented his loss in an editorial â but there were signals before his resignation that the seven-member Harvard Corporation was growing weary of his clashes with faculty. A call to the office of Summers' spokesman was not immediately returned Sunday. Summers' critics say the real issue was his confrontational management style, not his controversial comments or his ambitions for Harvard, which they say they generally supported. By the end, they insist, he had offended a diverse group of faculty. "There's a real free speech issue, but it's Larry squelching other people's free speech," said Daniel Fisher, a physics professor. "He's an incredible bully." But many supporters saw politics in Summers' departure. Law professor Alan Dershowitz has argued Summers was done in by a core group of faculty angered over his support for the military, Israel, and for his comments on women in science â the last of which he apologized for repeatedly. "I'm clearly in the left 20 percent of the country, nationally. I'm a Ted Kennedy liberal," Dershowitz said. "In the Faculty of Arts and Sciences, I'm in the 10 percent side of the conservatives. "That doesn't show I'm out of sync with the country," he said. "It shows how out of sync Harvard is." Right-of-center pundits couldn't agree more, at a time when some conservative students say they feel under attack in the classroom for their beliefs. The Wall Street Journal's editorial page wrote: "Only on an American university campus" would Summers "be portrayed as a radical neocon." Blogger Glenn Reynolds of instapundit.com predicted Summers' fall would help conservatives pass bills monitoring academic freedom â including one currently under consideration in South Dakota's legislature. More traditionally moderate to left-leaning media have also criticized Harvard's faculty. A Washington Post editorial said "professors, of all people should not require mollycoddling." Peter Beinart in The New Republic Online wrote Harvard's faculty "has just made an ass of itself." Yet physics professor Fisher says the contention Summers was the victim of thought police is a red herring. Most thought Summers' "outspokenness and willingness to raise difficult topics" was a "good thing," he said. "The main issues by far are the incompetent ways he has run (Harvard). He has run Harvard for the glory of Lawrence Summers as much as possible, not for the good of the university." Harvard, Fisher said, may take 10 years to recover. But others say it is the faculty that has smeared Harvard in the public's eye. Now, some people who may have donated money or sent their children to Harvard may think twice, said George Leef, executive director of the John W. Pope Center for Higher Education Policy, a North Carolina think tank devoted to what Leef calls "educational traditionalism." "What Summers was trying to do was restore some of the academic integrity that he could see and many other people could see has been eroding at Harvard," he said. "And for doing so, for saying some things that the faculty regarded as intolerable, he had to march to the scaffold."