Windows NT & Real-Time Applications: Solution, or Problem?

IACT member and computer software consultant John Drabik has authored one of our most widely-read 24x7 Reports, discussing the techniques and costs of using Windows NT in embedded, real-time systems. His report is a must-see for all business owners, CEOs, consultants, systems administrators and consumers now facing the decision to accept or reject Windows NT as an alternative to established network OSs such as Unix, OS/2 and NetWare.

If you've already read some of the numerous NT articles in print and on-line media, perhaps that media coverage disappointed you with its lack of objectivity as well as expertise. No such disappointments and dangers are waiting for you here! If you're ready to get the nitty-gritty facts about Windows NT, then you'll appreciate John Drabik's careful, well-balanced appraisal of NT's interface and underlying structure, of its performance level under mission-critical systems, plus its ongoing problems and limitations. Essential issues covered here include hardware requirements, security, Java, interoperability, essential compliance with standards such as TCP/IP, and responsiveness to programmers as well as users.

Copyright 1998 by John Drabik. First published at the IACT website, <http://pages.cthome.net/iact> on November 5th, 1998.

This paper focuses on a particular application area for embedded systems, called Character Generators (or CGs). These boxes are used to overlay the text, graphics, and images on your TV screen. If you watch sports scores pop up over live video, or see the name of someone being interviewed on the news, or even for some types of advertising, you are viewing output from a character generator. These systems have rather strict real-time requirements, and must accurately respond to vertical and horizontal video interrupts, for example. Multi-tasking is needed because CGs are tuned to creation of high-quality video output which is different from the output of typical PC video boards. As a result, it is usually necessary to create source material on the same box; otherwise, the on-air appearance fails to match the expected results.

Some manufacturers now produce CG systems based on NT, using off-the-shelf video cards. Such systems are far from matching the results of dedicated CG systems. However, as this paper progresses, you will see that many of the same factors that limit NT in demanding professional application areas, such as CGs, apply to other real-time or embedded systems too. This paper discusses Windows NT when used in such embedded applications, and why you should exercise extreme caution when considering such products.

At the end of the paper are links to other web sites, including some that address alternative operating systems, technical problems with NT, etc..

Contrast this with Windows NT, which will not even start with less than 16 MB RAM, and needs 128-256 MB for decent response. Expensive cache-RAM is required. Don't bother trying to start NT on anything but the latest, fastest processors either. And the disk-space "footprint" for NT is several hundred megabytes. NT and other Windows products are loaded with excess code, rarely-used modules, and cookies such as the MS-Excel flight simulator, a credit roll for MS-Office developers. These "features" waste disk space, processor resources, memory, and responsiveness. Where do they take you today?

Although many embedded systems use less RAM and disk space, and slower processors, they often outperform NT-based systems. This is especially true for specific targets such as CGs; as a result, such systems are far more stable, often running for years without a reboot. Their user interfaces are designed to work with artists, instead of leading them through a dizzying maze of menus, buttons, and inconsistent operations. Dedicated systems are designed to work in a symbiotic relationship with the operator, and not as a patch on a general-purpose box. With regards to performance, dedicated CG rolls and crawls do not "stall," as frequently occurs on even the fastest NT machines. (For character generators, a "roll" is when the text moves up or down the screen; an example is the credit roll at the end of a movie. A "crawl" is when the text moves across the screen; an example is that warning that just flew by from right to left: "Tornado in five minutes! Details at 10:00!") Such problems occur in other real-time systems too; motor-control, safety systems, and any applications that demand high utilization time are not good candidates for NT.

Since dedicated machines are designed as an investment in the target application, such as character generation, they include special-purpose hardware to provide effective resolution and compositing speed that are unheard of in NT systems. Rendering operations, graphics loading, and operator interaction are smoother, faster, and more efficient, because the machine is designed to meet those needs, not for game playing or word processing. On most dedicated systems, there is ample capacity for the material needed in production, without wasted space for cookies, poorly written code, or rarely-used "features." Smaller is not only adequate; it is actually better.

NT is designed for general-purpose computing, not embedded or real-time applications. Microsoft designed the kernel for MIPS processors; at the time of design, the Intel relationship was badly strained. Thus NT is not optimized or well-suited for Intel machines. Years ago, Microsoft dropped MIPS support, but the kernel was not changed (NT 5 is supposed to address this issue; industry watchers feel this will introduce 2-3 years of instability into NT). Microsoft invented the NTFS file system, although it suffers from the same performance and fragmentation problems as FAT-based drives. Microsoft also claims that NT has C2 security, but that is only for PCs in locked rooms, without a keyboard, floppy disk, or network card! It also applies only to the previous version (NT 4 is not certified). Such a box may have only limited use. At the very least, it is safe to say that NT has C2 network security only when it is not connected to a network at all!

NT is substandard in other areas, too. Multi-tasking is one; networking is another. NT Server (even on fast boxes with lots of memory) only handles about 200 network connections, and by the time you reach the 200 count, the system is all but worthless. OS/2 or Netware routinely handle 1,500 connections on the same hardware. OS/2 Server uni-processor outruns NT multi-processor in industry tests, yet uni-processor machines cost a fraction of what multi-processor systems cost. If you use NT in a production network environment to transfer art, text or data, the result is poor response, a need for additional servers, and increased risk of failures. The steady stream of "fail-over" products is testament to the fragility of NT.

Disk system operations are poor on NT. The RAID system is prone to frequent crashes and lost data. Floppy disks cannot be formatted while in the "My Computer" window, without stopping all other operations in "My Computer." IDE disks have far lower sustained speeds than the SCSI disks and "threaded" (multiple requests pending at once) drivers used in many dedicated systems, yet IDE cannot be threaded to improve performance; switching to SCSI on NT is possible, but NT has critical bugs in that area, which may result in system crashes and lost data. NTFS suffers the same fragmentation problems as FAT, and frequent unrecoverable crashes that lead to lost data. It is common for operators to reinstall NT and applications several times per year. If they're lucky, they run a backup before the associated data gets erased.

Are you hoping Java will help with NT interoperability problems? Don't bother. The Java Virtual Machine is incompatible, and runs at only 1/3 the speed of JVMs such as that on OS/2, according to InfoWorld [see link below]. While it is possible that the NT JVM purposely undermines Java, as some critics claim, there is a possibility that the problem is much deeper. Microsoft may be unable to build a decent and compatible JVM because of underlying system problems.

One possible reason could be the MIPS model on which NT is based. MIPS is a chip designed for workstations, and now found in game systems such as Sony Playstation. MIPS context-switching (between tasks, to handle interrupts, etc.) is done by changing a few registers; on NT, the cache must be flushed. On machines with a large cache (256 or 512KB), the results are slow. Adding cache doesn't help; in fact, it can actually reduce NT performance (more to flush).

Another possible reason is interrupts. Intel machines are interrupt-poor, a fact known to anyone who has configured a PC. Contrast this with hundreds of interrupts on MIPS, 680x0, etc.. Dedicated systems based on other processors easily provide many more sources of interrupts, and handle them more quickly. In moving the NT code from MIPS to Intel, this ability was lost.

Finally, MIPS is "register rich," while 80x86 architecture (Intel, AMD, Cyrix and similar processors, for example) is "register poor." So 80x86 boxes running NT waste time shuffling data, further slowing a system. NT is particularly hard hit because it is so bloated (so there is more code and data to shuffle). It doesn't help matters any when a prime design goal of the Windows '95 user interface, now in place on NT, was to add "cool" features such as animated icons and flashy graphics that do not contribute to solving any problem.

Thus NT is hit four times before it ever gets started-- in the cache, by interrupt limits, by data shuffling, and by a wasteful interface.

WOWEXEC (Windows on Windows Executive) is another layer, responsible for running many W3.x and '95/'98 apps. Anybody who suffers through a Windows3.xx app under NT, and then sees the same program fly on a system such as OS/2, is aware of this problem.

Internet Explorer adds yet another layer between the NT desktop and the user; soon, you may not be able to remove it. The penalty is high. It is common for IE to crash, fail to run Java apps or run them slowly, etc.. A web browser is an inappropriate model for dedicated systems, and restricts the ability to respond to real-time events. Will there be a new version of NT which removes IE and is geared to "real-time"? To believe this, one must not believe current Microsoft claims in court, that IE represents an integral enhancement to the system, and is not just an add-on. Or, the new version of NT will have to have a separate build structure, and thus will not be able to capitalize on the extended market of "standard" NT. There could even be incompatible NT applications, which would further limit the usefulness of such a special NT version.

Some NT-based, real-time product manufacturers claim that they moved to NT because customers demand "open" systems that adhere to industry standards. NT doesn't support many open standards, and actively undermines others:

1.) Sun claims Microsoft undermines JVM with incompatible tools and extensions; if your vendor touts Microsoft Java as an interconnection method, you could have trouble.

2.) TCP/IP, the connection "language" of the Internet, is modified in ways that greatly increase network loading factors, hindering performance.

3.) DNS, Directory Name Services, which is used to look up network connections, is thwarted by NT, which uses its own proprietary and incompatible lookup methods.

4.) Internet e-mail has been extended in ways that make communication between non-Windows machines difficult or impossible.

5.) ActiveX, a proprietary package, is routinely inserted into web pages and e-mail material in violation of Internet standards. Microsoft refuses to open the ActiveX source for porting.

6.) COM, Component Object Model, appears to have been hastily contrived to head off a powerful option called CORBA, yet CORBA is an open standard.

Is "everybody" going to NT? It does not seem to be the case. Sales are below expected levels, according to sales tracking firms. Many vendors do not provide NT-optimized support for devices, while others are reluctant to commit to NT due to a small installed base and the pending demise of most current drivers in NT5. NT doesn't appear to be a good option for sites with existing data, legacy applications, hardware that is adequate for a task but not for the same task under NT, and when BSOD (Blue Screen of Death) crashes might damage your reputation.

For systems like CGs, which must not miss rolls, crawls, or other critical video operations, an RTOS (Real Time Operating System) is the solution. NT is not an RTOS. An RTOS has features that NT can't match: close parameter control; fast interrupt handling; deterministic processing; accurate memory control, task switching, and resources. An RTOS avoids the layers of NT, Windows '9x, WOWEXEC, and NTVDM. An RTOS has smaller, tighter, faster code.

Hard real-time operation is the ability to accurately predict response to an interrupt, task switch, or other operation. It does not mean "if you need it to be faster, buy a new machine." Instead, the system is controlled so that performance is not adversely affected by loading. It implies a tight, highly reliable kernel that accurately responds to events.

NT is a varied assortment of interrupts, layers, and varying priorities in a model designed for general-purpose computing, on a different architecture from where it now resides. The model is inconsistent with hard real-time. In NT, it is considered imperative to give feedback like animated icons, to show that the system is responsive. Real-time systems ask, "What must be done? How can we guarantee response?" These are not compatible viewpoints.

NT is not equipped to smoothly run fast code. Such code exists in virtually all embedded high-performance products. For example, lightly loaded high-speed NT systems often cannot perform a CG roll or crawl properly, resulting in substandard video output. For other real-time systems, the result may be erratic or improper operation, an inability to respond to critical events such as safety interlocks, or an overly expensive "solution" to an otherwise simple problem.

Response on NT is, to a large extent, not under programmer control. It is difficult to bypass NT-applied dynamic priority modifications. Programmers are often forced to write their own incompatible memory management functions. And often, weeks or months are spent fiddling with message loops or thread communication, only to have that work destroyed when somebody installs another program.

Current device drivers may be unusable under NT5, since they do not conform to the new model. NT already has poor driver support, and may be thrown back another few years, according to industry watchers. If you buy an NT-based system, you will most likely need frequent upgrades and patches while Microsoft gets the bugs out. You may also need new hardware. Meanwhile, system problems may be commonplace; this could hurt your production processes.

NT4 has been out for years; compared to NT3.5, the biggest visible change is the inclusion of the W'95 interface. Under the covers, the size of the kernel nearly quadrupled, to about 16 million lines of code. NT5 purportedly has over 40 million lines of code in the kernel! NT code updates are infrequent, not because there are few bugs, but because the user base doesn't support the investment needed to fix them, according to sources at companies such as Diamond Graphics. And the sheer size of the kernel makes it a dicey proposition to consider changes to the base code. In fact, NT has some surprising major bugs, such as network, file, and record locking problems in databases, printing errors, and more. Any of these could affect your operations.

By industry accounts, NT5 will not be ready for serious use until 2002. Many vendors consider NT an "elitist" market-- they make more money in the consumer market, and do not design for NT. New application support is spotty, or a rehash of W'95 programs; many use torturous menus and overloaded toolbars, and are based on the bulky "Win32" application interface. New hardware has a limited installed base; vendors are slow to provide upgrades or replacement drivers for subsystems that are otherwise quite serviceable. And Microsoft delayed NT4, Windows '95 and '98, SQL Server, Internet Information Server, NT service packs, NT5 and other new code so often, it is unlikely that the situation will change soon.

Could NT5 meet the challenges? Possibly, but large new software systems have problems. In mission-critical applications, this may be a big gamble. If NT5 really is the answer to the problems of earlier versions, it must include major changes and thus, potential bugs; on the other hand, if it really isn't a significant rework, then it won't solve the problems.

A) Major architectural flaws

B) Flaws may result in driver and program breakdown in NT5

C) Performance is lower than alternative systems

D) Not designed for hard real-time applications

E) Weak in legacy hardware/software support

F) Subject to crashes, system stalls, and resource allocation conflicts

G) Violates standards such as TCP/IP, CORBA, and Internet mail.

H) The C2 security claim does not apply to the current version

I) The NTFS file system is problematic.

J) Suffers from a performance-robbing multi-layer architecture that lacks the hooks needed for critical applications.

K) Wastes resources on animated icons, mouse pointers, and glitz.

L) Requires more processing speed, memory, and machinery to approach the performance of dedicated systems.

M) Faster processors and more RAM don't address the underlying issues of suitability-to-task or appropriate design.

Until Microsoft succeeds in addressing this considerable list of factors, and NT becomes capable of effective and reliable real-time operation, caution should be used when selecting products that are otherwise better served by appropriate real-time operating systems and dedicated hardware and software tuned to the needs of specific markets.

Statements or opinions expressed herein are based on industry analysts' views as well as those taken from first-hand personal experiences. For any type of system or application, you should carefully consider all options. If you still feel NT is appropriate, then use it. But don't be afraid to say `NT? No, Thanks.`

• Real-Time Operating System Vendors

http://www.lynx.com
http://www.microware.com
http://www.windriver.com
These companies have info on why RTOS systems are increasing in number, not decreasing.

• OS2LCG

http://rover.wiesbaden.netsurf.de/~meile/los2cl.html
Shows large corporations that use OS/2, with some specifically doing so to get away from NT.

• InfoWorld's "Java server performance"

http://www.infoworld.com/cgi-bin/displayTC.pl?/980914analysis.htm
Consider the OS/2 part of InfoWorld's report on how good a Java Virtual Machine can be.

• "US Navy Dead in Water"

http://www.wired.com/news/technology/story/13987.html
This article discusses NT failures in a real-time system.

• Scientific American's "Rough Sailing For Smart Ships" from current issue, November 1998

http://www.sciam.com/1998/1198issue/1198techbus2.html
Alden M. Hayashi's article about Windows NT's suitability for naval ships can be found in the print magazine also.