The Daily Kos has a remarkable piece by Arliss Bunny (hat tip to Bruce Shneier for spotting this) describing the actions of the Fed on 9/11. The piece was researched by going through the annual reports of all 12 Federal Reserve Banks, and led the author to the conclusion that "on 9-11 and the days which immediately followed, a relatively small number of people did some genuinely, physically heroic things in order to keep the economy from going off the rails..."
When the planes hit the towers it was a busy day and all but one Washington based member of the Fed Board of Governors were elsewhere. Chairman Alan Greenspan was flying home from Switzerland at the time, and would not know what happened until his plane had returned to Zürich. It fell to the one senior person in DC, Federal Reserve Vice Chairman Roger W. Ferguson, Jr., to coordinate the Fed's response. In some ways Ferguson was the ideal choice, since he led the Fed's Y2K planning and response. That turned out to be a meticulously planned effort which did a good job of expecting the unexpected in ways that served them well. For example
During the Ferguson-led Y2K emergency planning it was determined that key employees should carry two cell phones from two different providers and pre-load them with a master list of phone numbers including their key co-workers, key bank customers, banking regulators, utility service providers and the like. In many cases they had multiple numbers for specific contacts at each organization. It was this kind of planning which proved invaluable at all the Fed Banks across the country. That essential people were able to reach one another was a significant cog in the process. For those that the Fed, in particular the NYFed, were unable to locate, industry organizations stepped in and tracked down their people linking them back to the Fed and freeing Fed employees to put out other (metaphorical) fires.
The story goes on to describe the dedication and courage of Fed employees (who were in physical danger in NY) aimed at maintaining liquidity in the financial system and public confidence in the markets when systems were down and normal market mechanisms did not function.
In the earliest steps, Ferguson and his team announced that the Fed discount window would remain open as late as necessary on 9/11 to meet the needs of all the member banks. While the NY Fed relocated to its disaster site in NJ, employees stayed in NY, in the shadows of the disaster, to keep things running until the backup was running. They also provided first aid and shelter to victims and a cafeteria and crisis center for first responders.
For example, back in 2001, the system ran on paper checks, which had to move between the 12 Fed banks by airplane. With the entire US aviation system grounded, this meant that checks were not moving or clearing. The check float, which was normally in the hundreds of millions of dollars at any time grew to tens of billions of dollars. Institutions holding the uncleared cash had to be forgiven the normal surcharges for what is otherwise an interest-free loan. Institutions waiting for uncleared funds were short and were given access to funds.
In general Fed employees all over the country worked together, and for long hours, to make sure the market had sufficient money (including paper money for bank branches and ATMs) to function, and that we ordinary citizens had access to cash so that we could eat and pay our bills.
These people were quiet heroes and we all owe them our thanks.
They also taught us important lessons about the importance of good disaster planning. Because the Fed had a great plan for dealing with communications and systems, their senior people could take some of that as a given, and focus on their day jobs of keeping the economy afloat. Imagine how much harder it would have been if they were scrambling to figure out how to communicate, how to recover systems, etc.
In early 2000 there was a lot of talk about how the lack of any disasters over the Y2K cusp had meant that it was all so unnecessary. I always felt that nothing happened precisely because of the hard work that we put into testing and prevention. It wasn't a waste, it paid off, but it's hard to prove a negative. The other byproduct was that in critical infrastructure areas such as financial services, the disaster planning that at least the larger players had done paid off.
There were glitches, of course. As all of the financials moved into their backup facilities, communications had to be established among them. As I recall, that was one level of granularity finer than the kind of streetwide testing that had been done.
The takeaway for your businesses is that good BCP/DR planning is not a luxury and can determine whether or not your business survives the next disaster. Expect the unexpected. Keep contact lists up to date. If possible keep your backup sites warm. The best way to ensure that a network is functioning is to routinely run data through it. If that's not possible, you need to turn it up and test it on a regular basis.
The story of the post-9/11 Fed is one of good planning, quick thinking, dedication, and heroism.
Thanks.