Go back

"Are You a Survivor? Or The Weakest Link?”

	Tales From the Crucible: Secrets of a High-Tech Expert Posted: May 07 2002 Warren S. Reid SYNOPSIS: This article discusses the increasing trend of litigation as a solution to failed systems projects. It presents the viewpoint and experiences of an expert in the systems planning, development, and testing arenas. The legal process is briefly introduced, as is the role of the expert witness. Then, three cases are presented showing: the systems and the parties; the complaint; the analyses, strategies, and opinions; and the verdict. Finally, some recommendations are shared that will help you if you happen to find yourself in court after an alleged failed systems project. The Future Is Here! Unfortunately, the future of your success as either a high technology user or a producer / developer of high technology hardware, software, and services, may not depend on the brilliance of your technical staff, or the speed of taking your product idea to market, or your global strategic alliances. In today's ever-complex society, Silicon Valley and business users are finding the newest high tech growth industry to be -- litigation! Your success today and in the future may actually depend on the skills of your legal department -- and not your other traditional production and service departments. The legal industry has been gearing up for the influx of new cases involving failed systems projects as well as intellectual property misappropriation. Organizations such as the Computer Law Association, the World Computer Law Congress, the International Computer Law Forum, the Software Publishers Association, as well as the International and American Bar Associations are devoting huge amounts of their resources and programs to educate constituents in litigating complex high technology matters. Jury of Your "Peers" The most disturbing prospect is that these cases all begin as a good faith effort between you and the opposing party to develop and implement successful systems solutions to benefit you both. Instead, you will now have to duke out the merits of your case under the worst circumstances, including: · Up to 5 years after the project went sour. · Spending weeks and months preparing for deposition, and gathering materials and readying yourself for the ordeals that lie ahead. · Anywhere from 3 weeks to 6 months at trial (although you won't necessarily be involved in all of that effort). · Having your every meeting, words, letters and memos, hallway discussions, facial expressions, and philosophies examined and cross-examined by attorneys, and judged by a jury. Another potential surprise and problem is illustrated by the Los Angeles jury process. Unfortunately, the majority of people in Los Angeles simply ignored their jury notices until recently. Stiff penalties and follow-up by the courts enacted only in January 2002 will likely halt this practice. However, juries oftentimes consisted of anyone BUT your peers – given the allowed self-selected citizen-withdrawal from the jury pools previously tolerated. The point is you can never be sure who will ultimately be on the jury to decide on your complicated technology issues that could span on-line, real-time, web-based, mission critical systems failures, systems development methodologies, satellite interfaces, enterprise resource systems, etc. However, I predict increasing disputes between vendors and users as systems become more complex, systems involve more integration with other products and services to work, and as the economy continues on its present course. The amount of time, resources, and energies spent in developing complex systems is simply too great and the impact of failed systems is simply too visible to let a failed projects go without a fight, without explanation, and (depending on the mindset of the parties) without retribution. Jury Selection A few more words about juries and the legal process is in order so that no reader glamorizes what may happen after he/she threatens to sue the other party -- and gets his/her wish. Jury selection is not a science by any means. As you have probably noticed in the O.J. Simpson trial, histrionics and drama can in some cases overshadow the truth. I'm aware of a situation where the prosecution's main witness, who was obviously losing the case and also had the most to gain, began to wail like a baby. While the jurors eventually found the company innocent of any fraud (after four and a half weeks of deliberation), they made a very strange recommendation to the court. They awarded a seven figure sum to the "wailer" -- in my opinion, simply because he broke down so thoroughly. My belief was confirmed in interviews with two of the jurors after the judgment was given. The result of any lawsuit is still somewhat unpredictable, and using the legal process is no substitute for being lazy and/or ineffective during the contracting, planning, management, systems development, testing or quality assurance processes. Juries Don't Understand Some believe that if they only get the right experts, they will win their case. In fact, some experts have been likened to prostitutes -- although I believe that is out of order. In my experience, any expert who really is or can be bought off (i.e., to testify to anything) is unlikely to ever get any more work and therefore the system weeds out such dishonest experts. What is more likely to happen is that the experts counter-balance each other -- with one expert swearing 'the glass is half empty' and another swearing 'the glass is half full'. Similarly there are so called "experts" who believe that metrics and function points are useless tools and others who believe they are imperative to the development, control, and management of good systems development projects. Sometimes, winning can be the issue of "which expert does the jury like more?” This is particularly distressing. For instance, an expert has a beard that is salt and pepper in color. If that expert reminds a juror of a deceased parent that the juror loved or enjoyed, such juror may be more likely to hear what the expert has to say and even side with that expert -- whether or not s/he really understands what the expert says. On the other hand, if that juror had a horrible relationship with that parent who had the expert's demeanor or looked like him/her, that juror may be unable to really hear that testimony or it may be uphill battle. Thus, the issue of going into court, to seek real "justice", can be a slippery premise indeed. While my experience shows me that the legal system works in most instances, there is a chance that, even if you are absolutely in the right, an acceptable judgment may elude you. Role of the Expert What then, is the role of the expert? In my opinion, it is to educate the judge and the jury, that is to try and explain the complicated systems development processes, technologies, interfaces, and requirements to people who very likely have much less experience than you would hope. While educating, the expert must use appropriate gesturing, speaking style, and dress for that particular courtroom -- else his/her communication with the jury might be impaired. Also, remember that most parts of the trial can be very boring to jurors. Establishing rapport and the importance of the expert's testimony is critical in the first 5 minutes of his/her testimony. Developing a strategy for getting the complex points across to the jury is a critical service the best experts provide. Another role of the expert is to explain the fact pattern, to put together the pieces of the mosaic so the jury can see what happened. Getting the jury to actively listen and "participate in the testimony" (for instance by saying "picture this" or "put yourself in the following situation" when answering the attorneys' questions) will improve the ability of the jury to put itself into the events in question and remember them better. Thirdly, the expert will opine on the issues described in the complaint, explaining in an independent manner what this set of circumstances means in terms of the contract, in terms of acceptable performance, in terms of the flexibility of the system, or portability or maintainability or fault tolerance, or whatever the issues are. In giving his opinion, the expert tries to help the jury understand from his independent vantage point and professional experience just what happened and what a reasonable conclusion might be. Lastly, the role of the expert is to assist and/or train the attorneys (with whom s/he is working) on the technical issues and questions that should be raised during the discovery process, while deposing witnesses, and in the trial. Competent experts tell the strengths and weaknesses of the case to the attorneys early on and at every step of the way. They have the courage to resign from the case when pressure is put upon them to flagrantly "spin" the facts or shave their testimony. Competent experts recommend to the attorneys that they are working with that they consider settling or dropping a lawsuit when the facts do not support the case from the expert's technical, systems, management, or business point of view. Of key importance is that the expert must carry out the above role with independence. He or she is not an advocate. To help retain that independence, experts are retained by the attorneys that represent the clients and not by the clients themselves. Furthermore, they are paid before they render their opinions so that their fees are not contingent on what they say or how they perform during testimony. Fabulous Failures With all of the new wonderful "panaceas" given to software developers and users over the decades to help assure successful system solutions, such as rapid prototyping, structured methodologies, JAD (Joint Application Development), object-oriented programming, test coverage analyzers, COCOMO models, function point metrics and software tools, project estimation and management tools, etc. successful systems still elude many otherwise successful businesses. Indeed fabulous systems failures include the following: · Chernobyl - where a series of "so-called" impossible errors happened in a particular sequence to create the most serious nuclear catastrophe ever. · Bank of America - where in Fe uary 1988, hardware problems caused the Bank to lose control of several billion dollars of trust accounts. All the moneys were eventually found in the system -- but all 255 people, that is, the entire Trust Department, was fired as all of the depositors withdrew their trust monies. · Atlanta, Georgia IRS - in the late 1980's when it had to pay tens of millions of dollars of interest to taxpayers on refunds because it was up to 9 months late in processing those income tax returns/refunds. · US Federal Reserve System - where the Fed's new system lost control over 28 billion dollars in transfers the first night it went live and thus awarded interest to the wrong member banks (Note: the member banks only reported $24 billion dollars in errors the next day - I wonder who the wise guys were?). · Tragedy at Sea - The shooting down the civilian airliner over the Straits of Hormous (the system was tested to be used in the open Mediterranean Sea, but it was actually used in the Straits where the pattern recognition software projected the incoming friendly plane as an enemy aircraft and, under the military rules of engagement allowed for the destruction of the aircraft -- killing all aboard). · Intel's Pentium fiasco - 1994/1995 · Microsoft's continuing delay of Windows 1995; Windows Me; Windows Office xp - Microsoft should actually be commended for this as they realized and have the courage to get the product out correctly, not just quickly. · Disney's Lion King CD Christmas Day 1994 fiasco - where its "much-ballyhooed" CD ROM would not work in thousands and thousands of homes much to the dismay of young children and frustrated parents. · Intuit's bugs in its Tax preparation software that allows for the preparation of faulty tax returns (it guaranteed payment of penalties for users that may be assessed such sums), and the March 28, 1995 defect that allowed outside tax preparers to accidentally gain access to and change supposedly secure tax returns kept on Intuit's data bases. The list goes on and on and will continue as other failures like the Denver Airport Baggage Handling System, California's Department of Motor Vehicles systems and other notable failures continue to come into public view. Let's not forget one of the worst system's disaster ever -- Jurassic Park. Although Jurassic Park is fictitious, you may recall that its system was based on the false assumption that only female dinosaurs existed in the park (so once the system counted to the expected number of dinosaurs -- it stopped counting -- even though new dinosaurs were being born in the wild), and that one brilliant but disgruntled programmer (Mr. Nedry) that provided no systems documentation was enough to keep the system running forever. Example Cases With this setting re high-tech experts' roles, and the fact that failures and runaways will continue to happen, I would like to present a few cases where I was engaged as an expert. In reviewing such highly summarized cases, it is easy for the reader to underestimate the amount of sleuthing and preparation required. Once you know the answer it's so simple -- but in some cases we have analyzed as many as 500,000 -700,000 pages of documentation to arrive at and support our conclusions. With two or more sides to every story and even more contradictions, it takes hard work and tenacity, along with clear, honest, and persuasive communications with judge, jury, and attorneys to win the day. "The Dispatcher-Car 54 Where Are You?" The System and the Parties This case involved one of the world's largest vehicle dispatching organizations (hereinafter called the "Dispatcher") and most complicated computer-aided dispatch (CAD) systems. This CAD handles thousands of different vehicles, covers a million miles, and requires a giant workforce, while managing special hazardous situations, assisting local police department dispatching and many other functions. It needs to "work" best and be absolutely non-stop during peak performance situations when natural or other disasters strike. The opposing party, the "Integrator", was a billion dollar plus consulting firm entering the CAD industry, that rightly believed that being the Integrator for this dispatch system for this entity would give it worldwide renown and instant credibility. After many delays in delivery and poor testing results, the Dispatcher canceled the contract with the Integrator for cause. The Complaint The Integrator sued the Dispatcher for over 100 million dollars even though the contract was only for a fraction of that size. The Integrator's claim charged that the Dispatcher's unreasonable contract termination for delay, faulty design, and inability to complete the project negatively affected their future business worldwide -- and thus caused the Integrator's loss of bids to develop CADs in Korea and Europe. Among other things, they sued for future profits lost. Such illogical and vastly inflated sums are oftentimes just the starting point for negotiation and settlement. But because such erroneous logic can sometimes elude a jury, these inflated demands must be taken very seriously from the outset and defeated. The Integrator further blamed its inability to finish the project timely on the Dispatcher's alleged continuous changes and unwillingness to freeze specifications. Despite this, the Integrator claimed it poured additional monies into systems development to responsibly keep the relationship and contract intact. It claimed to be on the "two-yard-line" with only a little more time -- weeks -- to deliver a fully tested and functional system. The Analyses, Strategies, and Opinions My team and I reviewed tens of thousands of documents during discovery. We interviewed Dispatcher witnesses and reviewed depositions from both parties. From this, we concluded: · The Project Was Horribly Underestimated - no formal estimating methodology was used (although Integrator proposed to follow Barry Boehm's COCOMO models). Furthermore, the system was erroneously architected and based upon closely coupling two large NCR computers, and a proprietary version of UNIX into a fully redundant, fault-tolerant, high-volume system. We believed the architecture was wrong from the beginning. The operating system and hardware internals were not geared to handle this kind of environment, but the Integrator said that they would create the necessary logic and interfaces. Without experience on how long something like this should take, the otherwise experienced Integrator grossly underestimated and understaffed that critical path part of the project. · The Integrator Was Unaware of the User's Needs - The Request For Proposal, like most RFP's was not perfect, in that there were ambiguities and alternative interpretations regarding required functionality. The Integrator claimed to be the expert in these areas (or promised to hire expert consultants in some of the esoteric areas). However such expertise was inadequately applied to this project, and many areas remained ambiguous or even unknown to the Integrator until they had a negative impact on the project schedule and dollars. We believed the Integrator was responsible to deliver the functionalities because they were manifestly evident or, as an expert in the field of CAD, the Integrator "should have known" about them. Burying its head -- or calling each requirement a "change" -- was no defense for not knowing. · Also, incomplete and inadequate communications between the Integrator and its subs resulted in holes in the delivered functionality, and non-preferred alternative ways of systems functionality. · Project Management Tools were Deficient - While the submitted work plans and work breakdown structure appeared to be adequate, the project staff did not charge their time properly to these task levels. Also estimates-to-complete were simply the mathematical difference between budget and actual -- with management never really managing or understanding what resources, time, or dollars were required to finish the tasks. Automated time and project management tools were of little value if the data was incorrect, late, or incomplete. In fact, having no system in that situation may be preferable as, at least, project management won't erroneously rely on faulty information. · The Staff Was Unfit, Inadequate, And Poorly Managed - We found letters showing the lack of a Chief Engineer and high turnover among Senior Software Analysts. We noted staff being pulled off of this job to work on other jobs that the Integrator believed were even bigger emergencies. · On December 24, one of the key testing managers quit the company. Associated with this event, we discovered what we hoped would be "a smoking gun" -- a personnel evaluation letter stating that the manager leaving was over her head anyway, without adequate training or support to handle a job of this magnitude, and that her loss would not impact the project -- although they didn't have a ready replacement. We enlarged this letter onto big blowups for our testimony with additional documents "proving" inadequate training, standards, management, etc. However, the non-plussed opposing side simply shrugged and responded that the letters were hastily drafted on Christmas Eve -- so the boss could leave for home. The boss was distracted at the time and "this sour grapes" letter didn't reflect reality about the departing testing manager. Remember, each side believes it has all the answers! · The System's Quality Was Poor - The lack of a formal or consistently used systems development methodology, as well as a particular lack of tools to prevent and detect defects before formal testing began, exacerbated (and perhaps caused) the poor quality. The Integrator's proposal actually promised to use Ed Yourdon's "structured methods" for certain aspects of the development. This was never done. · As the system moved through the life cycle, there were no quality review bars or exit tests to assure that the system was ready to progress to the next development phase. The lack of standards, plus ineffective walkthroughs during the requirements, analysis, and systems design phases caused the system to be pieced together in a way that made thorough testing of fail-over and other critical functions nearly impossible, and made cost effective and timely maintenance very difficult -- if possible at all. Subsequent testing proved this out. · Testing - There was no formal testing organization -- primarily just a group of self-trained testers. Worse, management was not committed to the testing group nor did it give it the power to delay progress through the life cycle or halt the release of a defective product. · The testing performed provided inadequate coverage of requirements. The Integrator's proposal promised compliance with Federal Regulations 2167A, and techniques outlined in Roger Pressman's seminal "Software Engineering - A Practical Approach." In fact, there was no requirements tracability matrix, nor a standard regression test environment and testbed. The Integrator's haphazard regression testing did not include re-running the code or the larger appropriate sub-systems as a whole to prove the fix worked -- let alone to prove that the fix did not cause defects in other parts of the system. Furthermore, their test plans and executions showed inadequate stress, volume, fault tolerance, quality assurance, and "bad data/boundary" tests. "Error Guessing" techniques were not used altogether. The coders created their own unit tests without input from others -- ultimately resulting in an abnormally high number of unit test errors to be found in later, higher-level tests including the integration, and systems tests. Such errors are more difficult and costly to isolate and fix when they are discovered so late in the life cycle. · Just 'Round The Corner - Lastly, the Integrator tried to tell the Dispatcher and the Court that a fully functional system was just around the corner -- in this case between 8-12 weeks off. We reviewed the design of the fault tolerant software in detail and Operating System enhancements, and had the attorney ask specific questions of the designers during their depositions. We concluded that the only way to make the system perform to requirements with reliability was to start all over again. Furthermore if their UNIX and NCR platforms were immutable, such an effort would take an estimated 100 person years of effort -- to be accomplished by a team of 30-35 persons over a three year period -- which was not too different from the time it took Tandem and Stratus to implement their two very different, fully redundant, fault-tolerant solutions. The Verdict The Integrator Company was an "employee-owned" company in a city then experiencing a harsh recession. While the local jury later told the attorneys that judging against and penalizing its own citizenry was extremely difficult, the Jury awarded not a single dollar to the Integrator -- and more than $1,000,000.00 to the Dispatcher to cover attorney fees. This was a stunning victory indeed! "The Good, the Bad, and the Ugly" The System and the Parties This second example is interesting because it started out as a copyright and trade secret infringement case and it ended up being won mostly based upon testing objectives. Sometimes you'll never know where the cases may take you. In this case, one party was a small systems applications developer an IBM Value Added Reseller (the "Developer") that developed, licensed and maintained vertical software for the wholesale distribution business. The Developer formerly an electronics wholesaler, developed his own systems solution when no appropriate ones were available. Eventually he enhanced and globalized it so that it was appropriate for the industry. Over time, because of his 25 years as an "insider" (i.e., a wholesaler electronics distributor) and a visible presenter at major industry conferences, his product became extremely robust with functionality and very popular. Now in his seventies, and with a terminally ill son, he wanted to sell the software company, but in such a way that his employees would have a place within the acquiring company. The opposing party (the "Acquirer") was a billion dollar systems developer and integrator that already enjoyed success with its products in manufacturing arena. It believed that having an outstanding complementary distribution product would critically complete its product line offering. It believed it was presently losing customers who wanted integrated manufacturing and distribution packages. The Complaint The Acquirer desired to get software with the same look and feel as its manufacturing package as soon as possible. Therefore, it went on the acquisition trail. After looking at many available systems, the Acquirer gave the Developer a multi-million dollar letter of intent to purchase the assets and software of the latter company with the right, in its sole judgment, to cancel the deal if after intense examination the software didn't meet the needs of the Acquirer. Preliminary discussions and visits had already taken place so that the Acquirer generally knew about the organization and the software, its history, its pricing and costs, its competitors, its functionality and capabilities. The Acquirer claimed it now needed to review the Developer's software code, technical architecture and structure in detail; approve its design; interview the customers, programmers and testers, trainers, implementers, maintenance staff, hot-line group, marketing staff, and sales and sales support staff; review standards and documentation; review and analyze the testing suite; etc. and then apply its own tests, to be sure that the system worked as promised and could be maintained by the Acquiring company. This all sounded reasonable at the time, especially since applicable law required that the Acquirer act in good faith -- even though it could cancel the deal in its sole judgment. This means that the Acquirer could not just change his mind for no reason, some immaterial reason, or for a fraudulent purpose. The letter of intent also provided that the intense review within 45 days. Bang! On the 45th day exactly, the Developer received a one-page letter that indicated that the Acquirer would not complete the acquisition because the system did not meet the performance requirements promised by the Developer and the profitability requirements needed by the Acquirer. In addition, the system could not be backed up in less than two hours using multiple tapes -- thus necessitating an employee to stay an additional two hours each day. There were two other small points in the letter of rejection. The Developer at first charged the Acquirer with copyright infringement because the Acquirer in a very short time came out with a similarly configured system. Later, after we arrived and based upon our opinions, the attorney also charged "bad faith" and "trade secret misappropriation" (i.e., Acquirer entered into contract with no intention to complete it in order to learn the successful Developer's secrets to best develop its own competing product) -- a very difficult thing to prove! The Analyses, Strategies and Opinions During our research we discovered the following: · Targeted Markets - The large company was targeting the $2-$20 million dollar wholesaler distributor for the system with a particular focus on SIC codes 50 and 51. It's mission was to be, "a leading supplier of such software, differentiating ourselves by superior service, product, and a singular vendor source; we are to be looked at as the high-quality supplier with justified premium prices...we are to understand the competitive situation in each market segment, and prepare an in depth analysis of leading competitors" Elsewhere they proclaimed, "We are to get our product to market in a record 11 months -- and not the 23-26 months that we estimate to develop this system in-house from scratch" This mission statement obviously wasn't contained in any of the design documents. We had to trace all the way back to the actual business plan of the company to discover it. The above facts proved to be quite important as we reviewed the performance tests conducted by the Acquirer. We determined that its highly "over-transactioned" tests (which included 11 order entry operators entering orders simultaneously) really support distribution companies with sales between $350 and $400 million in annual sales. This was confirmed by our own research and research performed by the National Association of Wholesaler Distributors re average order dollar values and number of line items per order for relevant SIC code orders in the 1988-1990 time frame. Fortunately the actual speed and timing of the transactions were logged automatically by a utility that was on the system during these "load tests". This enabled us be very accurate in our analysis instead of having to estimate the transaction rate and response times. Because the performance tests were actually for companies of orders of magnitude larger than what the Acquirer needed -- we concluded that the tests were, in fact, intended to make the system fail. With our analysis and opinion as their support, the attorneys for the Developer argued that the tests were performed in bad faith. This legal attack would not allow the Acquirer to hide behind these erroneous tests and avoid the letter of intent because of the results. We went on to prove that if appropriate loads were used, the system would have performed within the promised sub-second response time 98% of the time and never with over a 2.5 second response time (vs. the unacceptable 20-45 seconds resulting from the Acquirer's loads). · Backup Requirements - Information about the available peripheral backup devices, and the time it would take, were all available from either the Developer or from IBM before the letter of intent was signed. The Developer had identified such information in its Operations manuals. Therefore, we argued that the Acquirer knew or "should have known" that this system would take two hours to backup and thus could not use that argument against us later to void the letter of intent. · Motives For Bad Faith - Still, with all of the above proofs about the intentions and bad faith of the Acquirer, the jury needed to know why the Acquirer would go through all of this trouble and effort if it never intended to accept/acquire the software. What was the motive? We uncovered/developed several motives: 1. The ability to bring their product to market more quickly by understanding how certain complex algorithms worked 2. Learning how the best available system was designed and implemented (including the use of standards, technical architecture and design considerations, screen navigation, levels of reporting, module interfaces, etc.) 3. Adopting a suite of functionality that was studied and evolved by the Developer in over 17 years in the marketplace -- thus avoiding a long trial and error process in determining what current customers really needed and wanted (i.e., the Trade Secrets). 4. Learning the future enhancements that were slated, and their estimated costs and benefits · We discovered that of the 6 people that reviewed the system in excruciating detail, 4 of them went on to develop the Acquirer's own system from scratch. And while their original estimates indicated it would take over 2 years to develop such a system, in fact the system was running and available in slightly more than half that time. · Trade Secret Misappropriation - While it is very difficult to prove infringement of copyrights based upon similar functionality (because copyrights only protect the expression of the function, not the idea of the function), we were successful by employing unusual tactics. First, we argued that this particular set of available and future features and functions (which the Acquirer had access to only because of its letter of intent) represented years of effort, trial and error, and interviews with customers and prospects by the Developer in coming up with a product which, from a market perspective, exceeded all other similar products. And that this particular mosaic of functionality was indeed a trade secret of the Developer. Furthermore, by copying this suite of functionality, even though with a different "look and feel", the Acquirer would be able to ing its product to market more quickly, and win away customers and prospects from the Developer -- two things that make a trade secret so valuable -- and that could only be accomplished by having access to other developers and documents under false pretenses. · "Curiouser and Curiouser" - We were also able to bring into question curious similarities regarding how certain functionalities "interfaced with and were presented to the users of the system" when comparing the Developer's system and the new one developed by the Acquirer. While there are several popular and less popular ways to implement any of the "allegedly copied" critical functions, the Acquirer almost always did it similarly to the Developer -- an astronomically low probability for 12 complex functions if no copying occurred. · Meets Profitability Needs - Lastly, we also developed an analysis that showed that the Acquirer would be able to achieve the profit goals and margins because of targeted in its business plan for the new system because of the synergies provided by the Acquirer's manufacturing applications and the market position already enjoyed. We developed several launch, pricing, and maintenance scenarios to prove this point. The Verdict After four years of arguing this case, it was scheduled to go to court on one particular Monday. On Friday night, as I packed my bags to leave, the case was settled almost literally on the courthouse steps in favor of a happy Developer and his attorneys. "How Many Ways Can You Slice a Pizza?" The System and the Parties This complicated case was eventually fought in a Federal courtroom in Des Moines, Iowa. The system itself was an ingenious system that in fact would turn one of the world's largest pizza chains, Pizza King, into over 5,000 self-contained pizza manufacturing and distributing plants. It was a system designed to capture customers' pizza orders by phone and get them into the customers' homes ASAP -- where every minute counts! Its functionality includes: · Automatic Call Identification -- order takers are alerted immediately and automatically on screen as to: who is calling; preferences; last few orders; payment preferences and credit card/check info; directions to person's house, etc. · Simplified order entry process -- using multiple size icons for pizza sizes, toppings, and special preparation functions, allowing teenager operators to work fast and accurately. · Full-blown automated inventory system -- calculated actual, idea, and variance usage; automatic reordering from the chain's Commissary; keeping recipes, related costs, and profits for each item, etc. · Voice synthesis system -- which speaks to the pizza preparers telling them minutes left to meet delivery time guarantees, and special reminders such as picking up cans of Pepsi in the order (a common mistake), etc. · A fully automated pizza delivery system -- enabling drivers to "wand" in pizza departures times, the monies collected and dropped by drivers, reimbursable miles driven, etc. Driver instructions are in English or Spanish. Pretty neat! · Full Cash Control and Management Reporting -- including automatic nightly polling and consolidations by multiple categories, reporting by order takes, register, time of day, during promotional deals, and various cash balancing and audit reports at all levels of detail. This case involved three parties: 1. Party "A" - a developer for point of sale (POS) equipment and with clients including Denny's, Winchell's, Church's Fried Chicken, Burger King, etc. (The attorney for this party engaged us.) 2. Party "B" - also in the fast food POS business boasting clients including McDonald's and Taco Bell. 3. Party "C" - formed by a group of professionals and financiers from IBM, Big Six Accounting Firm, and MBAs from top U.S. business schools. Their role was to finance these systems at $700-$900 per month over 5+ years (most pizza store owner/operators cannot afford to pay $35,000 or more for a multi-register system that's integrated with back office operations). The Complaint During the development of the system by Party "A", Party "B" after a long and detailed review of "A's" system and assets, acquired Party "A". Thus "B" was now responsible to complete the system and develop a long-term maintenance capability. Party "C" was in a very big hurry to finance the systems and discount the paper to other financial institutions. Well into the project, "C" unilaterally decided "the system didn't work and would never work -- and that it was fatally flawed", and that it should be paid for $80,000,000 in profits it could have made over the next 5-10 years had the system been rolled out to Pizza King stores internationally. Party "B" decided to join with "C" in saying that the system "didn't work." "B" alleged that "A" lied to it regarding the quality and status of the system-in-progress that it acquired (even though "B" was certainly capable of performing such an independent audit and analysis). They figured that it would be easier to prosecute "A" and perhaps win a settlement amount than to defend the quality of the system currently in its second pilot testing phase at some twenty sites and perhaps pay no monies (or receive no monies as "C" was really a shell company with no assets). The Analyses, Strategies, and Opinions To win this case in court would require our defining what "the system works" actually means. We would also have to show that "A" didn't misrepresent (or worse) the facts to "B". More importantly, to win the case, the attorneys and we believed it would be imperative to get "B" over to "A's" side defending the quality of the pilot system and the development efforts. But how could we achieve all this? After reviewing the documents and learning more about this case and the roles of each party, we developed a three-point approach that would define to what extent the system "worked" by: 1. The Hardware Works -- Comparing it against the contract specifications for mean time between failure (MTBF) for the systems' hardware components 2. The Software Works -- Defining a metric against which the readiness and health of the system could be compared 3. Users Liked It -- Getting affidavits and confirmations about how the system worked from pilot store users and, if possible, from "C's" own internal documents The Hardware Works (MTBF) -- to prove this point we analyzed every single error report and problem report that the system experienced in the five months it had been pilot tested in the stores. We did this for every component at every store (such as IBM PS/2, a make-station printer, a network controller, a bar code printer, order entry terminals, voice stations, wall masters, network components, etc.) We eliminated from our analysis data involving "bad sites." The contract permitted this. A bad site was one that had an average MTBF of more than twice the average of all of the pilot sites including that store. This would eliminate special one-time burn in situations and poorly managed stores. We also adjusted the data for situations that would impact true MTBF calculations but should not be counted in our opinion. Further analyzing the true failure occurrences, we were able to prove that while the system did not perform adequately during the first 4-5 weeks in the field, it began to meet and exceed the required MTBF after that time. Once it surpassed requirements in week 6, it continued to improve -- never falling below the contracted rate again. What was so powerful was that the contract provided for more than six months before that phenomenon was required to meet the contract! Our challenge was to explain to a jury that while the MTBF for a particular component was 260 days, or 730 days, or 2,300 days, that the MTBF for the whole system of 22 components mathematically came to just 35 days per site. While the opposing side argued, "How would you like a car that needed to go to the shop every 35 days?" (something the jury could relate to), we had to explain and convince the jury that if one order entry terminal went down or one of the ticket printers was unavailable for a few hours, the system still worked and still met the performance requirements of the store -- and still allowed a pizza to get out of the store two to three minutes faster than with the old system (a critical requirements for performance). The Software Works -- to proving the software "worked" or was about ready for full production release was more difficult because there were no standards in the contract, and because all non-trivial systems have defects. What are acceptable defects and acceptable defect levels depends upon the specific case and the instant facts. What we did here is identify, by combining all relevant problem software logs (kept by "A", "B", "C" and in the stores), every software defect and its severity level for the 9 months of systems testing and pilot testing. We then developed simple graphics to show the jury that the system had reached the kind of "stable state" that we would expect before a production release. We charted software defects opened and closed for each monthly period; totals by month of defects opened and closed by severity level; average time to address/correct defects. We found that over time the defects found went from 90 to less than 10 per month; that severity of new defects went from "catastrophic" and "must fix immediately" in the early period to "deferrable" and "cosmetic changes"; that the defects backlog went from a high of over 100 in the early months to less than 10 per month; and average days to fix a defect was dramatically reduced from 84 days to 5 days. Combined with the facts we had already shown that the system contained the contracted for functional requirements and that quality control was adequate, these new statistics and trends proved to me and the jury that the system was stable and, just about ready for production from a software standpoint. "Users Liked It" -- Proving this was tricky - but luck was with us! During discovery, we learned that "C" was going to develop its own pizza system and, in our opinion, appeared to promise to use functionality that looked a lot like "A's" and "B's" system -- although they would recreate it in a different language on a different hardware platform. "C's" own business plan to raise money for this new business endeavor indicated how much current users liked "A's" and "B's" system and how the current users would be either references or the next buyers of "C's" new system -- a system which "C" would offer with more beneficial financing (based on transaction usage) to pizza store managers. "C" went on to praise the functionality, beneficial usage of the system, and their own experiences with "A's" and "B's" system to their bankers and shareholders as well. "C's" own project status reports indicated that many (not all) franchisees were complimentary about the system or its potential; and they included letters from users specifically praising the system -- although the system was not perfect! We discovered that no user rejected the system during the entire pilot test and that, even though the parties have stopped supporting the system after the lawsuit began, half of the customers continued to use the system daily for another 220 days to 352 days. This usage, in an environment where even a minute or two of delay or downtime can hurt reputations and guarantees of "half-hour piping hot delivery", proved to me and to the jury that the system worked. "The Turning Point' came, interestingly enough, after my deposition was taken for 4 days regarding the above facts and opinions, Party "B" fired its expert who tried to prove that the system didn't work -- and joined forces with "A" to win the case -- an important accomplishment. This made sense because "B" had actually finished up the system and performed over half of the pilot installations. Part of their arrangement to join "A" was that they be able to retain WSR Consulting Group as their own experts as well as "A". After the testimony above, "B" simply couldn't continue to argue that the system did not work. Now "C" would have to prove that alone. "It Keeps Going and Going and Going..." Just like that famous battery we further proved that "C" continually kept changing the system's requirements to get as much as they could from their relationship to "A" (and now "B") and refused to freeze requirements. We particularized 106 changes (some very significant) over and above what the contract required "A" and "B" to deliver. We further showed that "A" and "B" in an attempt to keep the project alive and "C" happy was issuing new enhancement releases of the software at the rate of two per month. The Verdict After 4.5 weeks of deliberation (so even with all of the above it wasn't an easy decision for the jury) the jury agreed that the system "worked" and they gave a stunning victory to the "A" and "B" team. What Can You Do to Prevent These Things, or Win in Court While there are no guarantees for winning your case, consider a following maxims for systems development projects that may very well lead to successful systems solutions and that if needed in a lawsuit may help: · Plan your project well, and manage according to the plan. Identify your assumptions and know when either they change, or the project profile changes. Use metric and lessons learned from similar and previous projects -- they could be your best guides. · Involve and commit users, vendors, and management throughout the life of the project. You will find as a vendor or a user, that the "three legs of the milk stool" that make for successful and robust systems development projects are users, and management, and the vendor(s). The seat of the stool is the good communications, contracts, and understanding of expectations between the parties. Without these things, they system is bound to fail. · Nothing is easy if you haven't done it successfully before -- especially in computers! All systems are not created equal. Whether it is a VisiCalc trying to develop a second product, or your expert database team trying to develop database software for fault tolerant systems -- all skills are not transferable. It is easy to underestimate the difficulty in implementing new technologies. Beware! Bring in appropriate expertise during the planning stage and have them on the project when they are needed. · Use the best "standard" systems development methodology (SDM) that you can find. If you do not, make sure the other party is aware of what you are using and "signs off" on how you are going to develop the system. Else wise, you'll find that you will be measured against the industry "standard" or the "best available methods" in any event, and that may put you in a very untenable position in court. · Estimate the time and resources adequately - not optimistically. It's amazing to me how many people say 'It's a $6 million project or it's a $10 million project! Let's get to work! Once we get started, we'll figure out how to do it! Heck, the budget is so large, we can even get lost the first $500,000 and then find our way." Sound familiar? Let me tell you, $6 or $10 million goes pretty quickly when: you have 50-100 or more staff on a project; when you have no specific, agreed-to direction or targets; or when you base your estimates and resources on unrealistic requirements, deliverables, expectations, and promises. · Deliver top quality deliverables -- as if a jury will review them 5 years from now. It is amazing to me to see the quality of many deliverables that I have reviewed. They are oftentimes inadequate, unprofessional, incomplete, and inaccurate designs, files, systems, analyses, and recommendations. What was done in haste, will surely come back to haunt you years later when jurors can take all the time in the world to deliberate the quality of your deliverable. Also, a deliverable may likely have more impact on jurors' decisions than a witnesses' recollection or a statement about the quality of the process. · Agree on acceptable performance, acceptance criteria, and acceptance tests -- before the project starts. It should be clear that developing a complex system could be the most difficult undertaking that an organization begins. Not having a clear and agreed to understanding of what is acceptable in terms of functionality, auditability, usability, recovery/restart, performance, documentation, options, etc. can most surely doom a project. If you don't agree on these things beforehand because there is no time, or in reality, because it is difficult to identify criteria or get parties to agree, it will be almost impossible later. And in court you will need this yardstick. This one is especially important! · Develop swift and escalating processes and procedures to address problems, concerns, and issues. Include these processes in the contract. Identify triggering events and resulting responses that will enable disagreements to move swiftly up the management ranks in both organizations and to mediation if required. The key for both parties to win is to stay out of court, and resolve problems so that the user can obtain the benefits of the system and the vendor can achieve the rewards of the profit margins. Consider binding arbitration as the last stop. In most cases it is less expensive than court; your case is likely to be scheduled much earlier; the judgment is binding (appeals are extremely rare); the selected arbitrators will have more experience in the subject matter than lay jurors. Setting up the appropriate arbitration clause is beyond the scope of this work -- contact your attorney before entering into a systems development project to get the right input. · Assure that the user understands what he/she is signing off. Be prepared to train the user if necessary. Simply getting the user to sign off on something that is incomplete, or the user doesn't understand (as for complex detail design documents, hardware configurations, or module hierarchies) will not necessarily hold up in court when it is shown that the user couldn't possibly evaluate what he/she was signing. This will hurt you, the user, and the project in the long run. · Allow contingency time for additional training, handholding, and problem resolution. Nothing hurts the ability and the teams to resolve difficult issues that unrealistic schedules and estimates that did not include contingency time to work out one of the most complex problems of systems development -- the communication between parties. Some Frequently Asked Questions Now that we have gone through some cases and cautionary recommendations that can help you stay out of court -- or win in court -- I'd like to share with you some questions I am frequently asked regarding how to approach large scale systems failures and relationships with attorneys: What should an attorney look for in an expert? The short answer is look for someone that you can work with. The litigation will have short deadlines at times, and will probably require several changes in strategy as more information becomes available. You need someone who can thrive in that environment. Look for people that can come to preliminary conclusions quickly based on incomplete information. Look for someone who has been through it before: i.e., from a technical standpoint, and from the legal process. Like a marriage or the hiring of a key executive, if the initial selection decision is wrong, a lot of therapy and grief will follow. Make sure the personalities are a good match and that the expert is: articulate; empathetic; loves to teach; can establish rapport with a jury (and judge); is credible; knows what he/she is talking about; is detailed; can simplify the complex; and can put words together into pictures (word pictures and graphics). Don't be lazy, check references! Note this is pretty much the same advise that I give experts re the homework they should do re attorneys that have approached them -- before they agree to render services! What do you look for when you start and engagement? In addition to the usual documents (chronology of events, names and roles of each key person involved, contracts, complaints, critical business and legal documents), large-scale systems projects have their own language and documentation. Such documents include the following (and this is by no means and exhaustive list): 1. Systems Development Methodology Used 2. Standards Used (in programming, design, testing, documentation, reporting, etc.) 3. Systems Requirements/Specifications (including functional, performance, growth, security, futures, etc. targets) 4. Project Feasibility Report 5. Systems Overview Chart 6. System Test Procedures, Plans, Results, and Discrepancy Management Reports & Statistics 7. Acceptance Criteria and Test Plan (with results) 8. Design Overview (and Preliminary and Detailed System Designs) 9. Detailed Work plans and Work Programs (staff estimates by task and actual staff Hours) 10. Automated Tools Used (for testing, capturing defects, documentation, estimation, progress reporting, coding, conversion, etc.) 11. Systems Conversion and Hardware and Software Installation Plans 12. All Deliverables 13. Various Inspection, Walkthrough and Review notes 14. Configuration Management Procedures, Tools And Utilities Used 15. Quality Assurance Plan 16. Project Management Reports 17. Contingency Plans It's the piecing together of these documents and many more, along with interviews of percipient witnesses, knowledge of the technology and/or specific client's industry, and knowledge of the computer industry's "standards of quality", where the expert can really make a difference! What one thing is critical to the success of the lawyer-expert working relationship? Open communications between the parties. The attorney needs to make and keep his needs known to the expert and vise-versa. As strategies change, and legal issues or positions change, the expert must be kept aware and contribute appropriately to those strategies and changes. The expert must be allowed to disagree and challenge the attorney's points of view where the technology, project, or industry records warrant such behavior. Keeping the expert in the dark re progress in the case that impacts the expert is a road to surprise -- and isn't the whole legal system filled with enough uncertainty already? How do you approach a large-scale systems failure job or other high-technology matter? Large high-technology projects, whether systems failures, runaway projects, getting a product to market, protecting or infringing intellectual properties, etc. are really a nexus of many factors (just like pieces of a mosaic) that I believe must be analyzed and pieced together wherever possible. While it is beyond the scope of this article to discuss all of the pieces and their complex interrelationships, the main pieces for a systems failure matter might include the items summarized below: 1. The Strategy Track · How does the system fit into on the long- and short-term strategies? · Of what strategic importance are these systems(s) projects and their success? · Of what importance is being on time? On-budget? On-target for the first release? · How much investment must really be made? How much will really be made? · What are the target markets? What can these markets absorb? 2. The User Track · Does the system meet User expectations? · Does the system meet specifications? · Is the system fit for the purpose intended? · Is the User interface appropriate (for the intended User)? · Are the interfaces to Company's policies/procedures and other systems appropriate? 3. The Project Management Track · Were appropriate management control systems and reviews put into place before, during, and after the project to: plan, organize, staff, direct, coordinate, budget, control, report, and re-direct the project as required? · Were progress and problem reporting and communications accurate, complete, consistent and fair throughout the project? · Were causes of delays properly analyzed, attributed and corrected and were systems put into place to prevent their recurrence? · Was the contract understood by the parties and modified as appropriate as the project continued? · Were estimates based upon reasonable assumptions? When the assumptions changed or when the appropriate party knew or should have known of such changes, were the estimates properly changed and communicated? · Did project management elect to spend all of the contract dollars before reporting that a system "could not be built as planned?" · Was the project appropriately staffed in terms of number and skills of staff, and expertise in the required tools throughout the project? 4. The Technology Track · Were appropriate architectures used to develop the system? · Were the final designs based upon appropriate constraints and future growth requirements? Were the compromises made reasonable? · Were appropriate plans made and executed to convert the data and the operations from the old system to the new system? · Were critical "-abilities" built into the system as appropriate: · Testability · Flexibility · Maintainability · Reliability · Usability · Availability · Portability · Understandability · Recoverability · Reusability (of components) · Was the system "ready for live production?" (A very complicated question!) 5. The Quality Track (Closely linked to other tracks) · Was an appropriate Systems Development Methodology planned for, and followed? · Was an independent (either in mind or in organization) Quality Assurance team employed? · Were adequate pre-test defect prevention techniques used (including appropriate inspections, reviews and walkthroughs)? · Was testing adequately budgeted, staffed, and performed? Were appropriate testing tools used (including: regression testbeds; coverage analyzers; store and playback systems; defect capture utilities; defect reporting and corrections systems; independent signoffs of critical tests, etc.?) · Were all deliverables delivered? Were they appropriate for the contract as modified by course of action (where applicable)? Was their quality adequate for the given situation? · Were appropriate "systems development metrics" kept and used? A Final Thought Judge Edenfeld once mused that “The jargon used in the computer field makes the legal Law Latin or Norman French seem as clear as the Ten Commandments … and while using exactly the same words, experts uniformly disagree as to precisely what they mean.” (emphasis added by author) Now there's something to wrestle with! ©Copyright, 1995 - 2002 All Rights Reserved WSR Consulting Group, LLC 4273 Noeline Ave., Suite 200 Encino, CA 91436 Tel 818-986-8842 Fax: 818-986-7955 E-mail: wsreid@wsrcg.com Website: www.wsrcg.com Go Top