Special Report from Tomorrow
I've had a few inquiries from some of you about my work for United Overseas Bank (UOB), and I thought it time to give a comprehensive picture of what I've been doing here for 1998.
Some History First
The project started over three years ago when the bank awarded a contract to IBM to write a system to computerize the paper process for making consumer and commercial loans. This went on and on without ever actually producing anything. As IBM dug into the real requirements, they realized how complex the job was. They made new estimates and went back to the bank for more money and time - a couple of times. Eventually the came back to the bank and concluded "it can't be done". All this put them in breach of their contract and after some finger pointing, the bank canceled the contract and did not pay anything. IBM was out a bundle.
One of the members of the IBM team was an Australian, Jeff Deluca, who I've known and worked with on and off since the early 1980s. Jeff thought differently about the project and eventually the bank offered him the task to build a team to produce a lending computer system. Jeff decided Java was mature enough for the task and if the model was done correctly, the software could deal with the complexity.
In 1997 a small project of 20 or so people was funded with the task to prove the technologies. Java was chosen as the language of choice as it allowed the bank to choose the hardware platform later. In 1997 the work was done on OS/2. Today the same work is done on NT and Unix.
Another key decision made then was to use Peter Coad's methodology. I hardly understand much about Coad methodology but one key aspect was to separate the business logic (PD, problem domain) from the interfaces (we have User Interface (UI) and System Interface (SI) components). Peter spent several weeks at UOB helping to design the original model. Part of the effort was to break the PD into small discreet tasks, called features, each corresponding to an object class.
By December 1997 the trial project was completed and they gave their demo to the Chairman. At the end the Chairman simply said, "OK", got up and walked out of the board room. "OK, what?" Jeff asked. No one knew. Jeff's boss said he'd ask in a few days. To the astonishment of his boss, Jeff picked up the phone and called the Chairman directly and asked what "OK" meant. The project was funded - he had the official go-ahead for the real project, Commercial Lending System (CLS), phase II.
About this time Jeff had offered me a job and we arrived in S'pore in January 1998. As I've written before, I'm manager of the support group. This is exactly the same sort of group I worked in at IBM for so many years - except this time I'm the manager. "Manager" in this context is not quite like at IBM, but more of a technical lead - except I do have personnel responsibilities, but no direct budget nor salary responsibilities. In part, this means I can hire or fire people, but don't have any influence on their pay levels.
My group is called "development support" in the org chart. We are responsible the infrastructure for the rest of the project. We run the Solaris machines (two then, now seven), administer Oracle (two servers now), administer Netscape's Suitespot servers (mail, news, LDAP and web servers), run the source control software (PVCS), install NT systems, deploy new/old hardware, run the weekly "builds", develop and maintain project status software and databases, develop whatever tools people need to work better and handle anything else that comes up (like demos).
When I first was introduced to people in the bank, my description of my group was, "We make them (pointing at the developers) look good. When things are going well, you can't tell it. When I'm having a bad day, everyone knows it."
I had five other people on my team when I arrived and within two months the first guy left - my Unix admin. In retrospect this was a huge blessing to the project because it allowed things to change. The guy liked to talk to impress people, but fundamentally he was useless. I took over the Unix admin role (yes, you should shudder :-) and started to whip things into shape.
The first job was to build credibility with the rest of the team. I told everyone in the project what our role was - to make developers work smarter and faster. We had identified the "customer" for our group. No changes were made without explaining what was to be done and why and ask permission of the development manager. We had a bunch of successes and people started to trust we knew what was going on.
Early on I harped on my team to understand what the job was -- to make the developers work better. Solutions had to be as automated and as simple as possible. I'd say, "I don't care what the problem is, the solution is a double-click of an icon - even better, they don't even have to do that." Time and time again I rejected solutions that started out, "The user installs...".
Then we had a disk failure - Holy cow, I hope those backups really worked! A very long day as I learned lots about Solaris and how support works and we were back up with about 4 hours of real down time and no data lost to our users. Another success - after a fashion.
Over the next few months I tried adding some folks to my team and within a month or so, they'd leave the project. I kept losing people and while I was interviewing, I wasn't having much success. Twice I "hired" people only to have them break off before getting them signed up. Once the guy had signed the papers with the bank and then backed out.
By late Spring 1998 the team was down to three of us. I did all the Unix and Perl-based (tools, build) stuff. Herman, our webmaster, was in charge of the Netscape SuiteSpot system (web server, mail, LDAP, news), and Mark was my NT, Oracle, PVCS and "extra" hand. In 1997 they chose PVCS as the source control system. In retrospect it was a very poor choice. It's taken six months for us to learn how to overcome it's failures - and this will continue until the end of the project. If you ever get a chance to use PVCS, skip the experience.
I wrote a bunch of NT Perl tools to make the developers lives easier. I also developed a Perl-based build process. Key developers "promote" certain levels of source code and once (or more) a week we compile it and it becomes the development base for the other teams (PD, UI or SI) to use. The build process continues to get more and more complex.
By May 1998 my team was feeling pretty low. I was terrified Herman was going to walk. I got the bank to send Mark and Herman on classes for SuiteSpot, so we finally had a clue how it works. In June and July I finally started getting people. First the Unix admin came and I suddenly had time enough to think about other problems. Then I got another tools person who has proved to be a real treasure. Gary's perfect for a support group - flexible, can multi-task, very clever and really really fast. He's a real hacker in the most complimentary sense. The last position was filled in early August - another web person to work with Herman.
Now in November, my job is so easy compared to six months ago. My team is now up to nine (I got the project office people too). No one has left in many months. I have time to plan and think about the future. Deployment of the application is now becoming the topic of conversation and my team will likely have a major role in that.
Team morale has soared. Each person is pretty independent and can largely take care of him/herself. I can now outline a problem and suggest a solution and say "show me something in a bit" and walk away. It's a great feeling. Someday I hope they'll get to the point of bringing me proposals for solutions to problems they see coming.
The "product" of CLS is a very large Java application called Powerlender. Early estimates were for 500 KLOC of Java. It'll probably end up at about half of that. The original IBM project was for about 60% of what we will eventually complete. By Sept we were just a little past the IBM "it's not possible" point - no small point of pride for Jeff, as you might guess.
Understand here, it's not that IBM is/was dumb, but the bank's loan process is astoundingly complex. Most technical leads are sure that once the bank starts using Powerlender, there will immediately be a barrage of requests to simplify the process, but right now, you can't get them to admit it doesn't ALL have to be done.
"Them" in this case are the CLS "users" - our project customers. This is a small core group of 8 or so people who do all the lending approvals for the bank plus another 200 people who will reference the information online. This core group handle your S$5000 loan application for a new motorcycle as well as a S$100,000,000 for a new taxi company in Indonesia. There are up to four levels of authorization required for a loan. In the vast majority of the cases, the Credit Committee (group of top bank officials) sign off on the loan.
A very high level description of the process for loan processing takes 3 hours of mind-numbing detail. A complete description takes about three full days to talk through. This is where the Coad methodology was so useful. Literally months have been spent by developers to understand what goes on here. Just the feature set for the problem domain (PD) will exceed 1200 features - each resulting in an Java Class. The two interface components, SI and UI, have defined another 200 and 1500 features to define their work.
Enormous energy continues to go into understanding requirements. The entire 1200 PD features is grouped into 25 or so larger topics. A feature-set is identified to be done, then the developers and specialists in the process go through the requirements yet again (this is usually the third or fourth time). Then Peter Coad comes from the US for a week and they talk through it again, and define the model. The object models and interfaces can now be drawn using TogetherJava.
Finally the 16 Java developers can actually start coding. The list of features to be done was chosen earlier in the process and the object models determine which classes are to be used (each class owned by one developer). This results in 3-6 "teams" working concurrently. Any given developer is usually on 2 or 3 teams at a time. The code is written and then reviewed by a group looking for errors. Finally, it is unit tested, merged into PVCS and each team tests the entire feature. When this is complete, the team leaders promote the code and it goes into the build.
It sounds like it takes forever - and at times the project seems to stall, then a group of features gets promoted and we surge forward. Unless you really know what to code, there's no point in coding and the process insures we get it right (as right as the requirements are).
Reporting on the Elephant
The bank has never tried to do serious development of this complexity before. Not unreasonably, upper management wanted us to report how the project was doing. Characterizing this in a simple form was a real challenge. Eventually what was developed (implemented by my team) was a database which tracks each and every feature for each domain and what step it is in (requirements, design, coding, inspection). When all steps are complete and the code is promoted, the feature is "done".
Once a week our project manager has a 15 minute meeting with the team leaders who report what features have completed what steps. This is entered in the database and color coded reports are generated and posted on the wall outside a top level VP in the bank. A one-page summary chart is made which rolls up the subareas into major areas and shows the percent complete. Each week upper management gets this plus a simple count of the number of features complete. The CLS users and upper management absolutely love the detail. No project has ever given them this level of feedback - and it only costs us a few hours a week to track (mostly time waiting for the color printers).
Much of this process will be familiar to IBMers, as this is not too different from the way code was developed in Rochester (no surprise, as Jeff worked there for several years). The emphasis on requirements is enormous, as it is the crux of the whole effort. This is as complex a system as I have ever seen. In some ways, it's more complex than even the operating systems I've worked with/on (at least I understood what was going on with the operating systems!).
Jeff has gotten a whole lot smarter than when he was in Rochester. He only recruits really really good people. He's convinced the bank they are worth it. He classifies programmers in these categories: A, B, C and gardeners. A "C" programmer is your average joe/jane who can do the job if you tell them what to do -- and watch them carefully.
Jeff only hires "A" and "B" programmers at a ratio of one "A" for every three "B" programmers. He claims "B" programmers do five times more work of "C"s and "A"s do an astounding twenty times more work. After seeing the caliber of people here, I believe it. Of course, since we have no "C"s, I can't be too sure of the work ratios, but his "A"s are stunningly good.
The three to one ration is important. One "A" can drive only so many others. "B"s are pretty independent, so they don't need much guidance. There's also the need to balance egos. :-) Jeff's been doing this for years and swears by the process. After seeing it here, I'm a believer too.
After so many years of not producing, much of the bank has been pretty skeptical about whether CLS would ever produce anything real. So in June 1998 we had to put together a demo for the CLS users (and indirectly our immediate management) as well as the Chairman. The effort that went into the demo was enormous and, while it was an outstanding success, it was terribly disruptive. Not only did it pretty much bring the entire project to a standstill, but it consumed at least a person-year of time getting ready for it.
It never ceases to amaze me, just how much upper management is in love with demos. Even if you explain how much of an impact it makes, how much development time is lost, they still want to see the latest version of the product, even if it isn't real. After the CLS users were stunned by how much was in the demo, the Chairman decided he did not want to see it after all - "September will be fine". The good part was that our immediate management, as well as the CLS users and the rest of the bank now believe we were for real.
So in September 1998 we were on the hook again. Yet again we paid the enormous price in developer and especially in management time. Once again the CLS users were amazed and pleased at what was there. The meeting with the Chairman didn't go as well. Yes, he believed the application would work, but there was something more.
Almost all loans in the bank get approved by the Credit Committee - meaning the Chairman is one of the users of Powerlender and he was vitally interested in how this worked and how it looked, and it didn't look like the piece of paper he has always had before, and he wasn't a happy camper about it.
So the political hacks went into "He said, they said" finger pointing mode. We went to work on yet another redesign of the UI. In retrospective, it wasn't very surprising. The Chairman is just like any other totally non-technical CEO who wants his new computer lending system to do all sorts of wonderful new things, but he shouldn't have to do anything different. It's really an issue of education - and taking the time to work with a key "user" (customer) that we'd missed - the Chairman.
This has been a really exciting project to work on. Java is real and can do serious applications. As an experiment, we took the Java jar files that ran on NT for the demo and ran them on both the Solaris machine and a Java Station - and both worked, although the JavaStation was far too slow. While it's supposed to work that way, it was still pretty amazing to see.
For the Powerlender project we also chose to use a tool called JRB (formally http://www.o2tech.fr). This is a very cool system. The Java classes are defined as "persistent". After the compile step, one runs a JRB import program which inserts calls to JRB classes at key points in the Java code (creating a new Java .class file). The result is that the Java developer does not actually code anything having to do with a database. Instantiated classes are just always "there" when you need to reference them. JRB uses a database to keep the objects for reference by Powerlender.
The use of JRB simplifies the design of the application enormously, but complicates other parts. Powerlender really isn't useful without a set of instantiated objects representing existing loans so it can show summaries of current loans and help make decisions on future loans. For instance, there are rules about how much of the total loans in the bank can be used for a given industry (like chemicals). In order to do that, Powerlender needs to make calculations based on ALL the loans the bank has made. If it doesn't have information on all the loans, the various limits will be inaccurate.
Deployment, the NutCrackerThe key to making a useful loan system depends on a set of objects (a database) representing 90,000 current loans. You can't really go "live" until these are entered. It'll take 3-6 months to enter the data. While you are entering the data, still more is being created and modified. You have to have those too.
So if you look at the schedule, Powerlender will be about ready in late 1999. This is not the optimum time to be deploying a project of this complexity. If you wait until 1Q 2000, you must continue to update the loan database all the time.
This is complicated more because one cannot look at the database by traditional methods - because the names of the tables and columns and their relationships is managed by JRB and doesn't really make sense to mere mortals. So while the code is being developed, the database schema changes as the model changes. How do you collect information on current loans? Where is the information kept? How do you enter loan information (e.g. instantiate the objects)? These are not easy questions to answer.
Powerlender (actually just PD) has hundreds of rules which will not allow you to create a loan that violates a rule - like "the discount rate may not exceed the brokerage rate" (whatever that might mean). So what do you do when an existing loan violates this rule? PD will not allow this to be put in the system - so you need some means to override the exception.
These are really tough problems which we are just beginning to look at. They've been largely ignored because until recently, top level management would not pay attention and allocate people to work on them.
There are several hundred potential end users of Powerlender in the bank. A small set of these are the principal users - the people who approve loans. Another group are loan officers at each branch of UOB on the island who need to see the loan information and occasionally make update (like change of address, change of status of the collateral etc). The last set of users are the members of the credit committee which includes the very top management in the bank (especially the Chairman).
Geographically this means Powerlender is used in 87 branches around the island, each connected by a 64Kbps link (which is also used for ATMs, tellers, and other bank business) and at the main headquarters.
Powerlender data is accessed very rarely compared to traditional transactional data. However, each "transaction" is gigantic in the size of the data accessed (measured in MB). Most data is read-only of course, with very little updating. Some is calculated (like totals) and can take minutes for the database engine to compute. So where is the Powerlender data to be kept? This is the key decision to be made for deployment. Is the data all centralized for easy administration? Will the 64Kbps links provide sufficient bandwidth for the large data being used? Will there be enough bandwidth if the links be upgraded to 128Kbps (a S$1,000,000 question).
Should the central database be distributed to the branches as a read-only database? What does it take to keep these remote databases in synch? What about the security issues?
In the Fall of 1998 all of these deployment questions have to be answered as they affect decisions on space, people, and hardware. The top level management, now being believers, are keen to have Powerlender out there now. It's particularly important to have loan information computerized with the deepening Asian economic crisis.
Some Lessons Learned
This is the first time I've really been in the "manager" role. I've been fortunate not to have many personnel problems. It's been "interesting" to be close enough to see some of the decision process being made (or avoided) at the management level.
This was my first time to have direct responsibility for evaluating people in a large organization. UOB's ranking process is basically unfair, but unchangeable. Now I know why all my managers hated the ranking exercise. How can you reasonably compare two people in different organizations, each doing different tasks, each with different management? And how can people who do not actually know these people make reasoned decisions? I don't think so either.
I've patterned my management style after two IBM managers I had who were most effective with their people (Debbie and Bob). I learned a lot at IBM about saying "no" when management begins to make arbitrary decisions (like setting dates for work that is not defined yet), and this came in handy here. By and large, Jeff has been very good to work for. We come from the same heritage and that helped immensely.
The bank is not in the software development business and despite the large investment in people and resources, the culture here does not really suit the software industry. There are lots of UOB policies which they try to apply "equally" to tellers and programmers without acknowledging there should/could be any difference. Many of the bank policies result in guaranteeing mediocracy -- forcing the really good people out as well as the really bad.
One mistake management at the bank constantly re-makes is to chose a date for a software task to be completed without defining what is to be done (requirements). The Powerlender project suffered on and off from this problem. In the initial stages, there was enormous effort to define the task (what do the "users" want?).
As the software project started to make significant strides, requirements got less and less attention (rather than more) and key people were not added or replaced when they left. Now as we near the end of the first year, most of the issues remaining are vague, not completely thought out, or just simply invalid.
In November 1998, almost all of the project was blocked waiting for some sort of requirement issue to be resolved. Each question asked and reasked, resulted in a different answer. One key area has been rewritten five times as the requirements were changed.
Finally, after getting the run around for months, Jeff made a decision to disband the requirements group and not ask for any more input. It was a way to force the project to completion. When the project enters it's final test in User Acceptance Test (UAT) next Spring, these issues will be resolved -- once. It's not the way software development is supposed to be done, but at least things can finish now.
So as I near the end of my first year with the project, I'm still very pleased to be here. I have been able to contribute enormously to the project and that is recognized by the people I care about. As I told my people in the interview, we have the best technology and best toys in the bank. Powerlender is the best thing going on in the banking business in Singapore (at least).
By the time I leave I'll be able to add a number of key skills to my resume. I'm no longer afraid of entry level management and can do a good job there, but I remain basically a techie. By the time I leave, I'll know lots more about Java, Oracle, Perl, LDAP, NT, Tk, Solaris, Marimba and administering a complex environment of mixed architectures. Seems like it should be useful to someone. :-)
I still go in every day with enthusiasm. I've got a long list of things to do and a seemingly never-ending list of surprises from upper management (which will likely increase as we get closer to deployment). But it's actually this sort of "invent as you're trying to catch-up" world that I am good at. I think I have excellent instincts about what to do in this sort of environment and I find it stimulating.
It's also not going to last forever. :-)