Untangling the VA computer crash

How Bay Pines hospital [in Florida] went from guinea pig to paralyzed victim because of pressure to implement a new system

ST. PETERSBURG - The new computer system was expected to bring order and efficiency to the sprawling Department of Veterans Affairs.

But the system failed a major test in late 2001 at a VA hospital in Fayetteville, N.C. Among the problems, and it had many, was slowness in completing basic tasks.

Months later, Bay Pines VA Medical Center was selected for further tests. Project managers wanted a larger, more complex test site.

During a dry run at Bay Pines last summer, managers cringed. The computer was still slow in processing transactions, from ordering supplies to paying vendors.

But the clock was ticking on the VA's prized project, already years in development. The pilot was scheduled to go live Oct. 6 at Bay Pines, the second-busiest VA hospital in the country.

On Oct. 2, the VA inspector general, which had been monitoring the program, warned that "several risks had not been mitigated."

In fact, an examination by the St. Petersburg Times revealed that warning signs occurred at several critical junctures in the project's development. In each case, the cautions gave way to inexorable pressure to get the system up and running.

On Oct. 6, the 240-member project team switched Bay Pines to the new computer.

The hospital still is digging itself out of a hole. Unable to keep surgical supplies in stock, it has canceled scores of surgeries. Staff can't keep track of finances. Vendors aren't getting paid.

After a series of articles in the Times, the pilot program, known as the Core Financial and Logistics System, or CoreFLS, is the subject of congressional hearings and multiple federal inquiries. A preliminary investigation by the VA inspector general has questioned whether CoreFLS will ever work, and how contractor BearingPoint got the contract.

At a briefing at Bay Pines on Feb. 5, hospital officials told VA Secretary Anthony Principi that CoreFLS is flawed.

"We have not realized any efficiencies since implementation of CoreFLS," according to the PowerPoint presentation obtained by the Times.

What's more, the system's future appears in question. The VA planned to roll out CoreFLS to 1,000 VA sites nationwide within two years. That plan is on hold, with more than half of the program's $472-million budget spent.

CoreFLS was supposed to help the U.S. taxpayer.

It was part of a movement that began in the late 1980s to make the federal government streamline its books.

The VA, the second-largest federal agency with 218,323 employees, lacked an integrated system for managing its $64-billion annual budget.

As a result, it's impossible to know where all of the agency's resources and inventory are at any given time, says William Campbell, the VA's assistant secretary for management.

CoreFLS began to take shape six years ago.

The Clinton administration directed the VA, which operates the nation's largest health care system, to upgrade its computer system by using software that was available in the marketplace.

Booz, Allen & Hamilton, hired as project consultant during the planning and acquisition phase from June 1999 to June 2001, submitted a plan for development of CoreFLS.

The system would use one database to perform a series of functions, from accounting to billing to contract and inventory management.

John O'Connor, the CoreFLS project director, said the goal was to "pass data back and forth very smoothly in a better organized manner."

CoreFLS would use three software packages: Oracle Financials, Dynamed and Maximo. BearingPoint was contracted to pull them together.

O'Connor said the software had been used successfully by hundreds of companies around the world. But the VA says that all three components never have been used together.

In what would become a familiar theme, an initial test during planning revealed major problems.

According to a VA history of CoreFLS, project managers discovered that the off-the-shelf software did not meet CoreFLS requirements.

The project team went back to the drawing board.

Round two offered little hope.

During the prototype phase, from July 2001 through June 2002, the CoreFLS project moved to the VA Medical Center in Fayetteville.

Nothing seemed to work.

"There were a variety of issues that came up in Fayetteville," O'Connor said. "One of the biggest . . . was the response time of the computer."

O'Connor characterized the problem as "a critical showstopper" that had to be solved before testing the system in a live environment.

VA officials blamed the slowness on telecommunications problems, though they declined to say how long it took the computer to respond.

After Fayetteville, however, "a revised strategy was developed to meet the new requirements," according to VA documents.

That new strategy brought CoreFLS to Bay Pines.

To work out the kinks, project managers wanted a bigger test site.

They needed a facility that was progressive, well respected and innovative.

VA officials in Washington settled on three: Bay Pines, the benefits regional office in St. Louis and the National Cemetery near the Central Florida town of Bushnell.

The first major test at Bay Pines occurred in March 2003. Had the system failed, the VA might have had to kill the project.

But O'Connor said it passed.

During a second test last summer, the results weren't as good. When placing a hospital order, O'Connor said, the system did not charge to the correct account.

In certain circumstances, the response was too slow. Congressional investigators say it sometimes took 50 seconds for the computer to complete a transaction that should take eight seconds.

O'Connor stressed that the glitches at Bay Pines - and throughout the life of the program - were "well within the norm" for a project the size of CoreFLS. The VA said St. Louis and Bushnell have not had such problems, though they note they are much smaller operations.

After a third test in September, a month before CoreFLS was scheduled to go live at the hospital, O'Connor said he was "reasonably convinced" the software was ready.

"In late September, based on data available at the time, the VA collectively felt that the prudent thing to do was to move forward," O'Connor said.

Not everyone agreed.

On Oct. 2, four days before the system was deployed at Bay Pines, the VA's inspector general said it alerted Campbell and O'Connor of "unmitigated risks" involving inadequate training, unreliable tests and unsubstantiated performance results.

Hospital officials say they pleaded with the project team to keep CoreFLS in the test phase while they worked out the kinks.

The system went live as scheduled.

"Day one," O'Connor said, "was very smooth."

That didn't last.

Sixty percent of staff did not have the appropriate menus to do their jobs, according to the Principi briefing in February.

And that wasn't the only concern.

The three different software systems in CoreFLS did not communicate with one another, complained Dr. Elwood Headley, the VA regional chief.

Marty Traxler, the CoreFLS project manager at Bay Pines, said the terminology CoreFLS uses is different from the old systems.

"It's as if we're learning to speak each other's language," Traxler said.

Even today, CoreFLS can take an hour or longer to respond to basic commands, workers say. What's more, CoreFLS is Web-based, meaning it is at the mercy of Internet connections.

For a hospital facing a staffing crunch, the news isn't good.

"The utilization of CoreFLS will require increased staff to maintain prior productivity levels in many departments," hospital officials advised Principi.

BearingPoint was prepared for the worst.

Before deployment, the contractor outlined a course of action in the event that CoreFLS failed and Bay Pines had to return to the old systems.

"Although unlikely, it should be apparent after the initial weeks whether CoreFLS is operating sufficiently to continue operations, or a catastrophic event has occurred which would require an immediate fallback . . .," BearingPoint wrote.

Catastrophic events would include inability to get supplies and services to veterans, inability to create surgery case carts, inability to maintain medical inventory, and inability to pay bills.

Since CoreFLS was deployed, every "catastrophic event" has occurred at Bay Pines.

As of March 18, Bay Pines had not paid nearly $2.4-million in bills because of computer glitches, according to federal investigators.

Angelique Rewers, a BearingPoint spokeswoman, referred all queries to the VA.

"Every major system startup has problems," the VA said in a statement. "When problems have been identified, they have been solved. None of the problems identified to date have been unsolvable."

The VA and BearingPoint blame the problems at Bay Pines on limited staff training.

Before switching to the new computer, CoreFLS users were directed to a Web site to learn how to use it.

It wasn't until November, a month after CoreFLS debuted, that the majority of hospital staff received hands-on instruction.

Margaret Davey, who works in surgical supply, said the training she received online was useless.

"It was totally different when we went live," she said, noting that CoreFLS menus online did not correspond with the ones on her desktop. "It wasn't what I expected at all."

At a congressional hearing at Bay Pines last week, VA executive Campbell announced a new round of training for CoreFLS users at the hospital.

About 100 people at Bay Pines use CoreFLS every day as part of their jobs. Another 600 use it occasionally. Within two years, VA's plan envisions training 115,000 CoreFLS users nationwide.

The VA says that in general, it will follow the training methods used at Bay Pines, but with more over-the-shoulder coaching.

Citing monthly payments to BearingPoint of more than $4-million, and with delays increasing costs, the VA inspector general says the VA needs to get a handle on the project.

Moreover, the inspector general backs warnings by hospital staff that the system may never work.

In recent weeks, a chorus of VA officials, including O'Connor, have acknowledged that it was a mistake to test CoreFLS at Bay Pines.

"Any time you disrupt services to your veterans," he said, "that's unacceptable."

Nobody seems to want to accept responsibility for selecting Bay Pines for the pilot.

During a recent congressional hearing in Washington, Dr. Robert Roswell, VA undersecretary for health, said he didn't know who made the decision.

But at the hearing at Bay Pines, Campbell said it was Roswell who recommended the hospital.

In an e-mail, Roswell later told the Times that he accepted responsibility for the decision to test CoreFLS at Bay Pines and apologized "if my testimony has caused confusion to you or your readers."

Congress, meanwhile, has signaled a loss of patience.

Last month, the House Veterans Affairs Committee requested an exit strategy, in case the system isn't fixed by May.

But the agency doesn't seem to have a backup.

O'Connor said that if the plug is pulled on CoreFLS, the VA will shop for new software and incorporate it with the working parts of CoreFLS.

Powerful members of Florida's congressional delegation are growing restless.

Last week, Sen. Bob Graham, the ranking Democrat on the Senate Veterans Affairs Committee, chaired the congressional hearing at Bay Pines.

House Appropriations Chairman C.W. Bill Young, R-Largo, has scheduled a hearing in Washington for Wednesday. He has invited Principi to testify.

At a recent meeting with Principi, Young and his colleagues expressed frustration with the way the pilot had been managed.

"We were definitely concerned," the chairman said, "about spending ($472-million) for something that doesn't work."

- Paul de la Garza can be reached at delagarza@sptimes.com or 813 226-3432.

COREFLS timeline

1990 - CoreFLS, a $472-million project, becomes part of a federal initiative mandating that government be more accountable and visible to the public by streamlining its books.

1998 - The Clinton administration directs the VA, the second-largest federal agency, to upgrade its computer system by using software that's available in the marketplace.

1999-2001 - The VA launches the planning and acquisition phase of CoreFLS, hiring Booz, Allen & Hamilton as project consultant, and BearingPoint as the systems integrator.

2001-2002 - During the prototype phase, the CoreFLS project moves to the VA Medical Center in Fayetteville, N.C. A test of the system reveals several problems, including slow response time by the computer.

2002-2004 - During the systems development phase, project managers bring the pilot system to Bay Pines VA Medical Center, the benefits regional office in St. Louis and the national cemetery in Bushnell.

March 2003 - A test at Bay Pines is successful, but a second one in the summer reveals more problems with computer response time.

September 2003 - Project director John O'Connor says he is "reasonably convinced" that the system is ready to go. He makes the decision to switch Bay Pines to the new system.

Oct. 6, 2003 - CoreFLS debuts at Bay Pines, despite warnings four days earlier from the VA inspector general that the system was not ready to be deployed. Almost immediately, because of software glitches, the hospital has trouble keeping surgical supplies in stock, forcing scores of surgery delays.

Feb. 5, 2004 - VA Secretary Anthony Principi gets a briefing at Bay Pines and is told that the system is flawed. Principi orders that the problems at Bay Pines be fixed before the system is rolled out nationally.

Feb. 13, 2004 - Bay Pines temporarily cancels surgeries as Principi and members of Congress call for an investigation of CoreFLS by the VA inspector general. Congress also launches separate inquiries.

March 2004 - The House Veterans' Affairs Committee gives the VA until May to correct the problems with CoreFLS, or submit an exit strategy, a plan on how it plans to pull the plug on the project.