One thing you never get used to as a politician is seeing yourself burned in effigy. It’s happened to me more than once, but the first time was in 2002 when I fought for a new program called Care Not Cash.

At the time, San Francisco had record numbers of homeless, and the amount of our cash assistance to them had gone through the roof. Each person was entitled to almost $400 a month — all you had to do was self-declare as homeless, then go pick up your money on the first and fifteenth of each month at any of dozens of check-cashing facilities in the city.

Our cash assistance was higher than that of any other county or city in California. No one even came close. San Mateo, just south of San Francisco, gave out about $60 a month per person. So not surprisingly, we had a massive migration of homeless into San Francisco — thousands of people who lived in San Mateo County would just walk across the county line to get that extra $340 or so a month. One woman who lived in Reno, Nevada, reportedly took a bus into town every two weeks just to pick up her cash.

This migration created a huge social problem. Invariably, on the first and fifteenth of each month, when the checks were handed out, our emergency rooms, sadly, would be packed with inebriated people.

Ambulances with other sick and injured people had to be diverted because hospitals had no free beds. Crime went up. The costs, both direct and indirect, were enormous — it was a debacle for the city.

San Francisco’s homeless advocates resisted any efforts to change the program, and at first I did too. I thought it made sense to provide people the cash they needed to live, and as a county supervisor I even supported increases in the amount of cash aid. But eventually, I couldn’t square my own liberal beliefs about helping those in need with my pragmatic desire to actually make things better.

The reality on the streets was that the $400 a month we handed out wasn’t improving people’s lives. What we needed was a plan to provide people with the care they needed, not just cash. No issue drove my passion more than homelessness, and, in many ways, it defined my two terms as mayor. It was the most obvious and visible expression of our city’s inability to solve problems, and I desperately wanted to do something about it.

So I proposed a program — Care Not Cash — that would cut the cash stipend to $59 a month and put the rest directly toward guaranteed housing and services for the homeless. People were outraged. Progressives and Democrats, nuns and priests, homeless advocates and homeless people were furious. But I truly believed, based on the evidence we had, that this was the better way to take care of the city’s homeless. It was pragmatism over pure ideology.

We fought for two years to implement Care Not Cash, through ballot initiatives and court battles. And when we finally were able to launch the program, what amazed me was how little hard information we actually had. We’d never done any real data collection on the homelessness problem, and because we had no data, homeless advocates and others could simply claim that things worked poorly or well based on anecdotes. They’d tell stories gleaned from the streets, tapping people’s emotions, and then demand that we make public policy based on those anecdotes.

This made no sense. We had no coordinated records system for people checking into shelters. There were no mechanisms in place for tracking people’s needs and fulfilling them efficiently. No one in government had any kind of usable, standardized data, so no one could connect the dots — people struggled with mental health issues, substance abuse issues, physical issues, but our efforts to help were scattered and slipshod. How could we solve the problems when we didn’t even have any concrete data on what they were?

The final argument for opening up data isn’t about whatever good things we can gain from it. It’s about the fact that opening up government data is just the right thing to do. We paid for it. We own it. We have a right to it.

A couple of summers ago, as a member of the Board of Regents, I was at a meeting for the University of California [UC] system when a spirited debate broke out about how to avoid tuition increases. The state’s budget had been slashed, and the financial outlook was grim. How could we bring enough money into the system so that students and their families didn’t get squeezed?

One of the regents suggested that we ramp up the licensing of research being done at UC research labs to our commercial contracts with the private sector, so we agreed to convene a work group on commercializing university research. Yet while I do believe we should make data available, there’s a gray area with this kind of research. I suggested to my fellow regents that this wasn’t really our data to sell: “It’s research done by public employees, funded by public dollars,” I said. “And now we’re going to monetize it like we’re a private entity? Who are we to do that?” I was concerned that we weren’t respecting the true ownership of the data.

The same thing holds true for all data the government collects, develops, or researches. All persons and agencies funded by taxpayer dollars are your employees. They work for you, and the information they produce is owned by you — which means they shouldn’t be able to sell it. Just as with the NOAA [National Oceanic and Atmospheric Administration] data, we can open it up and make it available for commercial use, as our default should always be to get the information out there. But open data means just that: open data, not data for sale.

As [Barack Obama’s first chief information officer] Vivek Kundra says, “We need a policy framework to shift data to the individual… The default should be that you own that information.” Kundra gets particularly riled when talking about how hard it is to get his own information. “The IRS [Internal Revenue Service] — it’s crazy that I have to go to my accountant to find out what I filed for the last seven years. Why can’t I get the IRS to tell me?”

But governments will always err on the side of releasing the minimum amount of data they can get away with. At a meeting with the State Lands Commission, which I chair as lieutenant governor, we discussed issuing a directive requiring the commission to release years of data — maps and information on parks, roads, and land use. The whole conversation was just push-back from the staff: “Well, how much are you suggesting we release?” And “We’ve already released x, y, and z,” and “We’ll get back to you with some ideas.”

It’s not up to us to “determine what’s appropriate.” If there’s a potential security threat, that’s one thing; otherwise, let’s put all the data out there. Public hearings, showing our meetings on public-access TV — these aren’t enough. We need crunchable, searchable data sets online, not foot dragging and obfuscation. These data sets are not ours to hide. They are the people’s to use as they wish.

As mayor and now as lieutenant governor, I’ve heard every possible reason from agencies as to why they can’t release data. “We don’t have the money!” Or “We’re short staffed!” Or “It’s in boxes, and we just moved offices, so we can’t get to it!” Or “The person who was doing this retired, and she’s the only person who knows where stuff is!” Or “Can’t we just wait until audits are done? We don’t have time right now for anything else.”

Behind the scenes, there’s always a mad scramble to figure out the absolute minimum that agencies can get away with releasing or how they might postpone releasing anything until we presumably get distracted and move on to something else. The foot dragging and excuse making are rampant. But why the reluctance? What is everyone afraid of?

In part, it’s just a holdover from history. Our government has never been transparent, and there was never, until now, any expectation that it would be. While he was a senator, Barack Obama was among the skeptics, according to Ellen Miller. “When he went to the Senate, he was the point man for post Abramoff election reforms,” she told me. “He held a meeting of reformers. I said to him, ‘I think members of Congress should post their official calendars online, so we know whom they’re meeting with.’”

Senator Obama’s response, according to Miller: “Have you always been a radical?”

“Since when is transparency a radical notion?” Miller wants to know. Apparently, for many in government, it still is.

And yet I can’t really blame anyone else for being reluctant to release data, because I’ve been reluctant to do so myself. While I was mayor, I told my staff I wanted to put my calendar online. Speeches, press conferences, public appearances, private meetings — I wanted everything out there. “Even your fund-raising meetings?” my chief of staff asked, his eyebrows raised. And that stopped me dead.

The truth is, you can’t be an elected official in this country if you don’t raise money. There are a very few super-rich candidates who can afford to fundraise less than the rest of us, but everyone has to do it. And the process itself is fraught. People don’t like the idea of their elected officials spending time soliciting funds, and the whole process, fairly or unfairly, feels unseemly. How would it look if, in a particularly busy fund-raising period, I had multiple private meetings with potential donors? Fund-raising may be a necessary part of politics in this country, but that doesn’t make it any more palatable.

And there are other people, not just donors, who also aren’t particularly happy about having their private meetings made public. As I immediately realized, it’s a lot easier to talk about total openness than to practice it. I wanted to walk the walk, but it would be difficult to do without causing harm to my office or to other people — people such as labor negotiators or business interests, who for their own reasons might not want to broadcast that they were meeting with the mayor.

In theory, I wanted to be totally transparent. But in the end I, too, compromised. Instead of listing meetings by whom we were meeting with, we listed them by topics discussed. In the interest of privacy, we made small changes in the description of meetings. These were cosmetic changes, and even writing about them here is my effort to be more open about them. But the bottom line was, I compromised. So I’m not perfect either.

Excerpted from Citizenville: How to Take the Town Square Digital and Reinvent Government. © Gavin Newsom, with Lisa Dickey, 2013. The Penguin Press. Excerpted with permission. All rights reserved.