Distributed Change Management Facility (DCMF) is a host-controlled software distribution and network management tool for DOS and OS/2 PC workstations connected to an IBM MVS mainframe. It provides facilities for the delivery and installation software into remote PC workstations; and for the management of PC software environment from a central location. This paper has two goals: to describe the system itself, and to servers as a case study of a local area network oriented software change control application.
Part I discussed LAN-oriented software distribution problems in general, and evaluate various approaches that address these software distribution problems.
Part II describes DCMF's benefits and how DCMF's data and function are divided among computers on the network.
Part III presents in more detail selected aspects of DCMF that illustrated novel facilities or implementation techniques.
Part IV summarizes the current state of the system and the lessons learned from it so far.
Until early in the 1980s the mainframe was the centerpiece of computing. Terminals and distributed processors were regarded as minor adjuncts to enterprise computing systems. Today, a radical change in perception had occurred. As the capacity of PC workstation hardware increased along with decreases in prices, people soon realized that these LAN-oriented systems could handle more than just simple file transfer application.
Now, LANs came on the business scene as a way of handling mission-critical applications with what was seen as a convenient multiple-user computing system. The downsized trend has started and is growing to replace high-priced central mainframe system with PC LAN-oriented client/server systems. The growth of distributed client/server network offers users some mix-and-match alternatives to the traditional mainframe architecture. Users are now discovering that LAN-oriented systems not only offer a definite price/performance advantage, but are dispersing computing power out to where it is needed as well.
However, downsizing is no plug-and-play affair. On large-scale LAN-oriented networks, the biggest problem faced by network managers are how to ensure the right software to be installed at the right place at the right time.
One example of this is Citibank's new PC application for managing Mexican bonds Citibank had mainframe systems equipped to handle bond transactions, but these systems were impractical for the new Mexican bonds transactions. Instead of wait for two and a half years to change the existing mainframe systems, Citibank decided to build this system on a PC platform. "This robust; LAN-based application is being used in manage $34 billion of bonds to Mexico's creditors, and it took less than three months to develop." ( See reference 12, Reddy, 1990, for details.)
Certainly, this application follows the downsizing trend and takes all advantages of PC workstations. But when Citibank started to develop the system, a major weakness on LAN-based application has been discovered. A mission-critical application that handles $34 billion bonds' transactions without a software change control methodology, how to perform ongoing application maintenance? How to ensure all PC workstations are running the same version of software?
The need to track changes in software has long existed. From a mainframe developer's perspective, the requirement has been to have source code match executable code, to pass executable code to testing, and to pass from testing to production. Each time source code is modified during maintenance, the steps must be repeated and the software synchronized.
Life was simpler when the mainframe was the centerpiece of computing. In a downsized trend, now that intelligence is installed on desk-tops throughout the enterprise, the requirement to control change extends to those desk-tops. The user of the intelligent PC workstation ranges from the experienced computer professional, to the non-computer literate staffs. Each may have similar requirements for control of their software and hardware configuration, and thus for electronic software distribution.
On a mainframe, software change can be tightly controlled because software resides only in the mainframe. Data center can establish procedures so only selected users, primarily systems programmers, can change programs. Where software changes are made in a data center under central control, a single change in a distributed network environment can affect thousands of workstations at multiple sites. In a client/server environment, controls are not as tight. Software resides on thousands of workstations as well as the mainframe. Almost any user has the authority to alter at least some portion of a program or a configuration.
When client/server and cooperative processing software architectures take hold, corporations face new challenges in managing the software distribution process. Rapid deployment of intelligent workstations, linked to day-to-day business operations, has spawned the need for more frequent updates to remote PC workstations. As the urgency of these updates, the issues of change management are becoming more urgent. New questions are emerging:
Currently, corporations rely on the following four approaches to manage these issues. They are sneakernet approach, mailnet approach, dial-up net approach and automated approach.
In "sneakernet" software distribution, technicians run around to workstations sticking floppy diskettes into PC hard disk to update software. For an example, The Travelers Insurance Co. wanted to upgrade an application that ran on 400 workstations. They had to send a team of technicians to four different locations and have them swap out software during the weekend. ( See reference 8, Korzeniowski & Desmond, 1991, for details.)
As the number of distributed workstations grow to the thousands, and the urgency of frequently software updates, this method become less efficient and more costly. Consequently, this approach is time consuming, inconvenient and expensive. Therefor, it will not fit into the client/server and cooperative processing software architectures.
In "mailnet" software distribution, developers put diskettes in the mail to deliver program updates and changes to remote production sites. Mail distribution of diskettes to thousands of locations leaves the task of updating software in the hands of often unskilled end-users.
When users are responsible for changes, other problems can occur. One user may not install new software, and instead work with outdated software. If a user works with a word processor, then it may not be important if he has version 5.0 or version 5.1. If the user users a mission-critical application and has outdated software, that oversight could cost the company a lot of money.
Furthermore, as the number of PC workstations grow to the ten-thousands, the method become unreliable. Not only was it labor-intensive -- ten-thousands of diskettes had in to be duplicated, labeled, verified and then delivered -- but the process often took months. It is obvious that this approach lacks the feedback and timeliness required for software updates. As more mission-critical applications have moved to PC workstations, along with more workstations have been added to the network, this approach certainly is not an alternative for software distribution.
In the more automated approach of "dial-up" software distribution approach, software transfer electronically. When the user requires a new version of software, he/she dial-up to a central location, and download the software from central host computer to his/her PC workstation. Saving the effort of duplicating and verifying ten-thousands of diskettes is the biggest advantage to this approach.
Since the software is installed by end-users, there is a risk when users forget to dial-up and download the software. They ran the risk of running outdated versions, and some lacked the training or skills to install the software upgrade. If a cooperative has to do four update per month and it takes about 25 minutes for each workstation, this approach gets very expensive. For a 20,000-user network, four updates amounts to 33,000 hours, or 200 staff-months that would need to install the upgrades.
Network performance is an issue too. When all users within the network want to download a time-critical software concurrently, the network will soon exceed its capacity. Also, there is no way to know which workstations are current and which are not. For an enterprise that has more than 20,000 PC workstations within its network, the dial-up approach for software distribution still is not an alternative.
Since sneakernet, mailnet and dial-up net are not capable to handle a large-scale enterprise LAN-oriented network, automated approach will be the only alternative to be used. In fact, using software to install software automatically in a timely manner is becoming more popular. Today more software products are being developed to enhance this technology. Automated software distribution involves network-wide delivery of changed or updated software to PC workstations on multi-user LAN-oriented environments.
The technology provides for unilateral synchronization of updates for the client and server's applications. Also, networks are assured that all users are utilizing the same version of software. Areas covered by automated software distribution include file transfer, software distribution, change management and network management issues. There is no doubt that the automated software distribution method is the best approach to be used in a large client/server and cooperative processing environment.
One existing system that used this automated approach is Distributed Change Management Facility (DCMF). DCMF is an integrated software delivery system that controls the distribution, installation and management of PC workstation software from a central IBM mainframe facility. It is an effective tool to control a LAN-oriented network systems' workload and overhand costs. It also is a sophisticated and cost-effective way to distribute mission-critical application software from one centralized location to ten-thousands of file severs and PC workstations throughout the enterprise.
Distributed Change Management Facility (DCMF) is a PC software change management tool, which developed by Bank of America's network systems management group. Today, Bank of America uses this DCMF to manage its Customer On-Line Information Network (COIN). This network consists over 20,000 PC workstations in 1,500-plus branches and administrative offices.
DCMF is useful for the Bank to manage this sophisticated communications network. It enables the bank to download PC software from a central site simultaneously to all branches' PC workstations. New version of business application and operation software can be installed overnight on schedule without any human intervention at the site. It's a real help in standardization and software version control. Without this DCMF system, the bank would require systems programmers to go out and manually install and test the software at each site for each workstation.
In 1986, Bank of America started its branch automation project. The project installed LAN-oriented PC workstations at branch offices, and designed software that linked the remote PC workstations to IBM mainframes. When mission-critical applications have been moved from IBM mainframe to workstations, the bank needed tools to distribute applications to branch offices and then manage the installation of various programs.
After failed to find any vendor that was close to delivering the type of product the bank needed, the banks developed its own change and software distribution system. Distributed Change Management Facility (DCMF) was initially used in a pilot program to automate bank branch offices. It ensured that branch office users were working with up-to-date software, and helped central network technicians manage remote connections. DCMF now runs on about 1,600 LANs and supports more than 20,000 PCs at 1500-plus Bank of America offices.
DCMF runs as a CICS application on an IBM MVS mainframe. It handles network management control for configurations, software distribution, operation status and problem management.
While all PC workstations are interconnecting local-area networks, a centralized integrated network management system is highly critical to police LAN activity. With DCMF, a network system administrator can identify PC workstation hardware configuration information to determine if there is sufficient disk space for a new release of software. Also, a system administrator can define distribution and installation schedules, and authorize software and data transmissions.
The complexity of client/server environments means problems happen a lot faster. This makes remote problem determination and problem source identification essential. With DCMF, a central system administrator can identify workstation software installation status and network communication status. When a remote problem occurs, a system administrator can send down commands from central site to diagnostic the remote network problems and fix the problem remotely.
DCMF is designed to simplify the complex job of managing electronic software distribution. A key objective is to cope with constant change and to implement it smoothly and without disruption. Not implementing change is tantamount to putting the company at a competitive disadvantage or out of business. Implementing new products or software carelessly is disruptive, may waste money, and can spur a loss of productivity or business effectiveness.
The design philosophy adopted by the designers of the DCMF was to provide a controlled environment for enterprise networks, and to standardize PC software releases enterprise-wide. For every PC workstation within the enterprise network, DCMF controls the following:
Major components in DCMF are Central Distribution Facility (CDF), Software Upload Facility (SUF), Intermediate Distribution Facility (IDF), and Target Distribution Facility (TDF).
The Central Distribution Facility (CDF) is an IBM CICS/MVS-based application. It allows central network system administrator to specify network PC workstation configurations and define the packages of business software applications that will be distributed and installed. The Software Upload Facility (SUF) is the single point-of-entry for the PC workstation software files that will be uploaded into the DCMF host data base for subsequent distribution.
The Intermediate Distribution Facility (IDF) is a remote component of DCMF that resides in a PC workstation at the user site. It serves as the client's software distribution node. The Target Distribution Facility (TDF) is the remote component of DCMF that resides in each PC workstation. Its role is to install new versions of software files received via the LAN without requiring any user action or knowledge.
Prior to the installation of a new software package into the enterprise network, system administrators install, configure and test the package on the system administrator's LAN. Upon completing the system testing, the package will be staged onto the file server of the upload PC. A network system administrator defines the software version and setup distribution destinations at the host computer.
Once the software and distribution have been configured, DCMF will collect all newly defined software files into a host software repository. A software installation command will be sent to each Software Distribution Node (SDN) within the network, to direct IDF to participate in a software distribution process. The Intermediate Distribution Facility (IDF) receives software distribution commands and collects the versions of software files from the host software repository. Those files will be installed on a configuration of targeted PC workstation on the LAN at the site. This minimizes data transmission requirements, since software files need to be sent only once to a single system at each site. DCMF subsequently distributes the received software updates to the PC workstations served by its LAN.
The Target Distribution Facility (TDF) will invoke the installation process. It uses a date and time that has been pre- defined by the network system administrator and forwarded by the IDF. This means that new software version can be installed during non-business hours, making the change process transparent to end users. The TDF will report the installation status to the central host computer upon completion of its processing activities.
To handle software distribution in a large-scale enterprise LAN-oriented network automatically, network performance is a major consideration. DCMF fully utilizes the intelligence of remote PC workstations. Tasks scheduling, files collection requesting, collect time pacing, and software health checking are all performed by remote PC workstations. Software download window can be configured, so that high-volume software transmission is done during non-business hours.
A data compression feature ensures maximum use of an available bandwidth. DCMF integrates data compression techniques for faster file transfer. Software files are compressed at the upload PC workstation before they are transferred, and then decompressed when they are received, prior to the software installation. The compression reduces network traffic by shortening the software file down load time required for file transfer. It also reduces LAN traffic and file sever utilization in a fully populated LAN.
Fidelity, the error rate of the network, is another key performance criteria of enterprise network. For many image transfer applications tolerate error rates, but banks' data services do not tolerate errors. As more mission-critical data is pushed out to client/server applications, security becomes of paramount concern. Moreover, a software distribution system cannot have any data errors. Without strong safeguards, an automatic software distribution approach would increase network vulnerability to hackers and computer viruses.
To prevent all of this, at software upload time, check-sums are calculated for each file to maintain software integrity. Virus monitoring systems also be implemented in the upload PC to ensure all software are installed by DCMF are virus free.
In addition, DCMF provides a software auditing tools. Network system administrator can issue software audit commands to any PC workstation within the network. A detail report of auditing results can be seen at the central site. In the case of audit failure, such as some files have been altered or damaged, the fixing command can be issued from the central site to fix the problem. DCMF then reinstalls all software again and corrects the problem automatically.
Beside the software files can be altered or damaged, the enterprise network has many points that may be failure during the software distribution processing, or during the day-to-day operation. DCMF has built-in some sophisticated functions to manage and control damages in its communications network.
One DCMF feature executes file transmissions. The software will suspend a transmission if a network problem occurs, and automatically resume the transfer once the problem is solved. A screen display illustrates distribution status and indicates any problem. Whether users are concerned with restoring files after information is accidentally deleted, recovering from the loss of software program due to equipment failure, DCMF has provided many methods to help manage LAN data.
If it is determined that a file is missing, or damaged at the workstation level, DCMF triggers a file recovery from the file sever. If the file is not available on the file server, DCMF initiates a file recover task with the central host and recollects the files. This ensures that all software is consistent.
The system need to probe deeper into the inner workings of file server and PC workstation, for example, in order to ensure that no single program in the PC system is responsible for bringing the entire network to its knees. No matter how well configured to prevent trouble and provide adequate change management a network is, there is a potential for trouble. When trouble comes, Problem Determination Facility (PDF) can be used for troubleshooting and problems fixing.
PDF is able to send command remotely to any workstation within the network. The most frequently used commands include remote IPL the workstation, display directories, and communication diagnostic remote PC workstations. From a central console, a network system administrator can send and retrieve files, generate reports, monitor software information and fix a problem. By using PDF commands, network communication and LAN diagnostic programs can be executed remotely, and the results will be sent back to the central sites.
One of the greatest advantage of DCMF is that people at various levels of support can use this PDF as a tool. Everyone who needs it has easy access to comprehensive network diagnostic and troubleshooting information. With the file retrieve files function, as the problem is forwarded along to application developers or systems specialists, all the retrieved information is also forward. The effect is streamlined diagnostic analysis and a significant reduction in duplicated efforts. Without PDF, network support staffs have to call an on-site LAN administrator to collect diagnostic information. They have to request off-site systems specialists to analyze the problem, and send technician to the problem site to fix the problem.
LANs are now being utilized in downsizing efforts in order to support mission-critical applications. A system for integrating management of local-area networks into enterprise network system is very important in these efforts. DCMF is an integrated system to manage the downsized LAN-oriented software distribution. This system is not only can be used by the bank, but also can be used by any enterprise that wants to handle an IBM network with 10,000 to 30,000 workstations.
Bank of America opted to sell the DCMF marketing and distribution rights to IBM. In the fall of 1990, IBM began offering the product as Distributed Change Management Facility (DCMF/MVS). Although IBM already had similar host-based product, Netview Distribution Manager, but it doesn't have the easy of use to handle an environment with over 10,000 workstations.
Automated software distribution touches on file transfer, software distribution, change management and network management issues. IBM sees electronic software distribution as a subset of change management. Change management encompasses more than software distribution, but distribution is a big piece of it. Today, IBM expects to use DCMF as a "tactical solution" for large-scale LAN-oriented software change management. For future, the functions of DCMF will eventually be incorporated with NetView Delivery Manager.
Traditional distribution methods, such as "sneakernet," "mailnet," or "dial-up net," do not provide the synchronization, completeness, and accuracy. The automatic software distribution and network management tools, DCMF, presented here are making it possible for network system administrators to say good-bye to those traditional distribution methods.
DCMF has successfully demonstrated at Bank of American that bulk software installations and upgrades can be accomplished quickly and efficiently. There is no doubt that DCMF has brought a new sense of tempo and harmony to the distribution of software at the bank's COIN network. DCMF provided the bank with the competitive edge to survive -- not only during a recession.