Google and Healthcare IT

Google is showing interest in becoming the provider of people's electronic health record. It's an interesting idea, but the ramifications of the U.S. Patriot Act are probably unacceptable to the vast majority of people who think about the privacy of their health record.

Basically, the Patriot Act gives the U.S. Government the right to look at any data in any computer in the United States, and they don't have to tell anyone they're looking at it. In fact, it's a crime to tell anyone their data was looked at. This basically violates any privacy legislation in any country that has such a thing. You need to give permission to anyone to look at your data, and you need to be informed if someone looks at your data for any reason.

In healthcare IT here in Canada we now live with the fact that healthcare data about Canadians can't be stored in or even pass through the United States. At least we can build an electronic health record. We just have to keep the data in Canada. Thanks to the Patriot Act, Americans may miss out on a great chance to improve their health.

(There might be an opportunity for enterprising Canadians to host the U.S. electronic health record, but if I recall correctly HIPAA says you can't store U.S. healthcare data outside the U.S.)

Anti-Economies of Scale

You must have noticed that IT is full of situations where the economy of scale is actually negative. Unlike retail, the bigger you get the more per unit everything costs. For example: An off-site backup costs nothing when a trusted employee can just take a tape (or jump drive) home. No need for special backup networks or LTO-3 tape drives either. You store your data on the cheap hard drive in your PC, and disaster planning consists of having another commodity PC available to which to restore your backup. This can all pretty much be done by any MCSE-type person that you can hire on a per-job basis.

Contrast the preceding with a larger enterprise: You have to pay Iron Mountain to pick up and store your tapes, tapes that you create from some complicated backup implementation. You need experts for the network, experts for Exchange (you need your own e-mail server, don't you), experts to run the SAN. You need to put this all in a special room, something that has its own design challenges. You may believe you need experts on staff who know your environment to handle all this. You can't just hire commodity skills on an as-needed basis.

The advantage goes to the manager who identifies what can be handled as a commodity, and either purchases the service outright, or at least organizes the internal staff so that the "commodity" tasks and responsibilities are done by "commodity-skilled" people. This has some interesting ramifications that I'd like to address in another post.

Why Agile Software Development isn't More Prevalent

I believe that agile software development leads to better software faster and therefore cheaper, and I think there's a lot of evidence to support that. So why, some seven years after the Agile Manifesto and many years after people started to do agile development, aren't the practices more common?

One factor that I think can't be underestimated is that agile pushes software developers and their managers out of their comfort zones way too much. Most development managers are more comfortable providing estimates up front and then reporting for most of the project that, "everything seems to be on track." At the end of the release cycle you have a period of discomfort, but that discomfort is mitigated by all the effort being put into getting the team to "pull it off."

Compare that to having to say early in the development cycle that, "we can already see that we're not going to get all the features in time, so do you want to drop features, and which ones, or do we move out the date?" That's way more uncomfortable because it's likely to be met by, "No to both." Then what do you do?

For developers, it's much more comfortable to just code what the Business Analyst told you to code, without having to address whether the user will deal with it. Moreover, why would a programmer want to produce code faster and cheaper? They get paid no matter how much work they do. Why would they want to even measure speed, as that could lead to being compared with their colleagues? It's much more comfortable to not even look at how fast the programmers are going.

Of course, successful agile organizations address these points. If you're looking at trying to make a non-agile organization more agile, you need to consider how really big a task you're setting yourself. It's hard to get people to change if they're not motivated to change, and the people who have to change to become more agile are actually motivated to do quite the opposite.

Talk About Missing the Mark

I just read an article in the Risks Digest that contains a quote that totally misses the mark:

The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry...-- Henry Petroski

I'd say that software is barely starting to approximate the needs of the user, and if hardware can't keep up, well then those hardware engineers better get their act together and start producing something useful. We treat Moore's Law as some kind of miracle, when in reality all we're seeing is what can happen if you let smart people solve the same problem over and over again for thirty years. They get better at it. We should be more surprised if they couldn't double processor speed every 18 months.

The real challenge of course is, as my brother quotes someone, "not what the program does, but what the user does." Or in this case, what the user can do. If the hardware doesn't support a productive user interface, then it's no more than a mildly entertaining magic trick.

Eggs and Baskets

One of the patterns I've seen over the years is how IT and technology in general magnifies the consequences of poor performance. We put everything on one storage device so it's easy to manage, but it's also easy to type "drop table..." in the production database. It's also very hard to get the backup just right, and harder still to be able to test restores of a production backup.

As an IT manager, you need to mitigate for this. Do reviews, walkthroughs, or whatever you want to call it to ensure that at least two pairs of eyes have evaluated your designs. Do as much testing as possible. Fight for more budget for testing. Build the simplest possible solution that meets your needs.

Another thing you can do is subscribe to the comp.risks RSS feed and read all the different ways that the best laid plans can be derailed by one little mistake.

NEC Storage Devices

It's interesting how NEC's press release on their new storage devices spends so much time talking about how they do non-disruptive upgrades and scale across the whole product line. We had trouble with upgrades and scaling our storage at Vancouver Coastal Health. The NEC press release validates that we're not the only ones. I don't have experience with NEC storage devices, so I can't comment on them. I'm only commenting on the press release.

Test Everything - Yeah, Right

I just read a short article that said the first thing to do to solve a lot of our IT infrastructure problems is to, "Create and maintain a test environment that mimics the production environment. Also commit to testing every change." (Emphasis from the original article.)

No problem. Since my 2,000 square foot data centre is already full of the 450+ physical and virtual servers I have in it, I'll just build another data centre and buy another 450 servers with all the associated software support and licensing costs. The lease costs alone on 2,000 square feet around here are about C$40,000 per month. I'm sure most IT managers have enough spare change in their budget to pull off that one -- not.

On top of that, you can never test everything. When I led the virtualization of 135 physical servers, the only problems we had after going live with any of the virtual machines were caused by changing IPs (which we had to do because of the enterprise network design). We wouldn't have caught those with a test data centre. There are lots of problems similar to that one lurking in a typical IT infrastructure.

You've got to build flexible infrastructures. Make sure nothing uses hard-coded IPs. Make sure your DNS doesn't have legacy pointers hiding in corners that you forget to update. Limit the partitioning of networks by physical location as much as possible in the first place. Build commodity server infrastructures.

(P.S. We did have other problems virtualizing the 135, but we found them before we turned the VMs over to production. The only problems that were seen by users were IP-change-related.)

Business Impact of IT

A friend of mine, Don Gardner, sent me a link about the importance of communicating the business impact of IT projects. The article says we IT people too often report the facts (e.g. we installed an e-mail server), rather than what that does for the business.

I liked the article, but I found it a challenge to apply to the projects I was working on at the time. Part of the challenge was that my projects hadn't been designed with business impact in mind. I was simply supposed to implement some bits of IT infrastructure.

Because of the complexity of IT we have to break our work into pieces. The pieces of work are very large in terms of technical complexity and cost, but vanishingly tiny in terms of their visible connection to business needs. We IT people get the connection, but it's not obvious to anyone else.

Change Management and Project Management

I heard a good talk on Friday that identified for me a real tension between change management and project management. That wasn't the point of the talk. The presenters were just presenting change management in the context of a major health care IT initiative that's underway in BC. However, their words of guidance to project managers triggered some thoughts.

To avoid confusion: We're talking here about "change management" in the sense of how to help people change the way they work, typically with the introduction of new technology. We're not talking about change management as in the management of changes to IT infrastructure itself.

The presentation showed the "Six Human Needs" that you had to look at when undertaking major workplace changes. First on the list was the need to be heard and have control. To give people control means they might change what the project has to do. The tension is that our standard project management approach in IT hates scope change. We have mechanisms to handle scope change, but we certainly don't encourage it.

I think there are a lot of situations where this sort of creative tension actually leads to a good result: The change management people will bring good ideas back to the project team. The project manager will push back due to budget constraints, and the customer will get the "best" result.

This puts a lot of pressure on the scope change decision-making process. You have to get those trade-offs between scope and budget mostly right. In my experience, it's been really hard to get the right people to take the time to truly work through an issue and come to a reasoned conclusion on that type of issue.

I think this is an area where the IT industry's "best practices" need to be improved. As a project manager, you're pretty much between a rock and a hard place when you get the change management input. If more projects would formally recognize the exploratory stage that most require before confirming budgets and scope, we'd be a lot better. Not being a PMP, I don't know if the Project Management Institute (PMI) has done a lot of work on the quality of a change management process, as opposed to the existence of one. I'd be interested in hearing what they've done.

By the way, the talk was part of BC Health Information Management Professionals Society (BCHIMPS) spring education session on Friday. The BCHIMPS spring and fall education sessions are excellent meetings for anyone involved in health care IT in BC.