Microsoft Internet Information Server Performance Analysis

March 1996

Business Systems Division
Microsoft Corporation

Abstract

The following paper analyzes the performance of the Microsoft® Internet Information Server in comparison to other Windows NT™, UNIX®, and NetWare®-based Web Servers. This analysis covers both performance of static HTML page serving, and dynamic Web page serving via CGI and Microsoft's Win32® extension for its Web server called Internet Server API (ISAPI). Microsoft's Internet Information Server, now an integrated part of Windows NT Server, provides Web-site managers with a Web server that's easy to install and manage, and provides both high performance and a platform for a new generation of Web applications. Microsoft Internet Information Server easily outperforms other PC Web servers by as much as 400%, and even outperforms more expensive UNIX/RISC-based Web servers.

Note: The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

Contents

Executive Summary
Introduction
Why Is Performance Important for Web Servers?
Performance Analysis

NSTL Benchmarks
Conclusion
End Note
Appendix: WebStone
References

Executive Summary

The key aspect in which the Microsoft Internet Information Server (IIS) most dramatically differs from its competitors is performance. In addition, Microsoft IIS is the only Web server integrated with the Microsoft Windows NT Server operating system, and thus derives a wide array of benefits for its users. Installations running IIS find its ease of use, scalability, portability, security, and extensibility superior to any of the alternatives. To demonstrate the overwhelming performance advantages of IIS, a series of benchmark tests were run by independent testing laboratories. The results are compelling and absolute: no matter how the tests were run--using different workloads, a variety of competitive Web servers, multiple operating systems (including UNIX) and even running some tests on expensive, proprietary hardware platforms--Internet Information Server delivers up to 500% of the performance of all of the systems tested.

The performance of a Web server is critical. Rapidly increasing communications speed, the use of multipurpose servers, the delivery of dynamically generated information and the explosion of internal Webs ("intranets") combine to make the need for high-performance Web servers mandatory. Only the Internet Information Server delivers the absolute performance today's sites require, on a platform that is easy to install and manage, scalable, and secure and extensible by using a similarly high-performance application programming interface (API), called the Internet Server API (ISAPI).

Please visit http://www.microsoft.com/intdev/ for more information on developing Internet applications using Microsoft tools and platforms.

Back to table of contents

Introduction

This document is designed to give a performance comparison between the Microsoft Internet Information Server (IIS) and other Windows NT, NetWare®, and UNIX®-based Web servers. Three general comparisons were made; two were new benchmarks run by independent laboratories and one is a comparison of publicly available performance information from a variety of hardware and software vendors. The two Web server performance benchmark tools that were used includes the National Software Testing Laboratories, Inc. (NSTL) Web benchmark tests, and WebStone version 1.1 from Silicon Graphics, Inc. (SGI). The NSTL tests were run by the National Software Testing Laboratory at their offices, and the WebStone tests were run by Shiloh Consulting and Haynes & Company at the offices of Shiloh Consulting. The sources for the cross-platform comparisons are WebStone results published by different hardware and software vendors.

Since there is not a standard metric for publishing Web server performance, the graphs in this document depict different performance characteristics of Web servers that the benchmark tools measure. Note that some of the tests are configurable--whenever appropriate, the configuration of the test run is supplied for reference. In all instances the conclusion is consistent: Microsoft's Internet Information Server running on Windows NT Server delivers the highest performance available.

Back to table of contents

Why is Performance Important for Web Servers?

Until now, Web servers did not focus on performance as an intrinsic design premise. This was acceptable during the early adopter phase of the Web, as most Web sites serviced a limited load over low-capacity links and less demand was placed on the servers. Today, the feature set for Web servers has stabilized, commercial implementations are available, and the importance of Web server performance is increasing. Several reasons that contribute to the increasing focus on performance include:

Back to table of contents

Performance Analysis

The WebStone version 1.1 benchmark was used to generate the workload and measure the performance of the Web server. All tests were conducted by Shiloh Consulting at their facility. For complete details on the WebStone test configuration, please refer to the report from Haynes & Company and Shiloh Consulting "Performance Benchmark Tests of Microsoft, Netscape, and Novell Web Servers," which can be found at http://www.microsoft.com/infoserv/.

Shiloh tested various Web servers, including:

For these WebStone tests, Shiloh used a Hewlett-Packard® NetServer LS server, with an Intel® Pentium® 133MHz CPU, 32 megabytes of RAM, two 1-gigabyte hard drives, and a DEC™ 10/100 megabit Ethernet card. The one exception is that for the BSDI UNIX Netscape™ Communications Server tests, 128 megabytes of RAM were used. When only 32 megabytes of RAM were installed with BSDI UNIX, the server repeatedly failed to complete the tests and the WebStone tests could not be run to their conclusion. Even with 128 megabytes of RAM, the WebStone tests were not able to complete the tests at the 128 client load without the server crashing. The other Web server tests on this platform were run with 32 megabytes of RAM.

Back to table of contents

Analysis

A wide variety of characteristics determine the overall performance of computing systems. Web servers are no different in that respect. The tests that assess the relative performance characteristics of the Internet Information Server and contrast them with the same attributes of competitive products, measure the widest possible array of relevant statistics. In these tests you will find analysis of the following:

For details on the results see the complete Haynes/Shiloh report at http://www.microsoft.com/infoserv/.

Back to table of contents

Throughput

Throughput measures the maximum rate at which the Web server transfers data to its clients. Throughput is reported as megabits per second. For reference purposes, the capacity of an ISDN line is 64Kbps (or 128Kbps if two channels are used), a T1 line is 1.54 megabits per second, and a standard Ethernet has a capacity of 10 megabits per second. Many local area networks now take advantage of new technology, such as concentrators, hubs, switches, or FastEthernet media which is capable of delivering data on the order of 10-100 megabits per second.

[IIS1822C 5382 bytes ]

Figure 1. Throughput

As the graph clearly shows in Figure 1, Microsoft Internet Information Server is capable of delivering as much as four times more data to clients than the Netscape Communications Server version 1.12 on Windows NT Server, and almost three times more data than Netscape on a BSDI UNIX. When compared to Novell's Web server NLM, Internet Information Server is more than 50% faster.

The capacity for high performance--that is unique to the combination of Windows NT Server and the Internet Information Server--is mandatory for today's Web sites. The additional capacity and capability extending from a high-performance architecture includes:

Back to table of contents

Connections per Second

Connections per second represents the sum of successful interactions across all the clients. An interaction consists of creating a TCP/IP connection between the client and the server; sending a request from the client to the server over this connection; receiving the correct data back from the request; and closing the TCP/IP connection. A higher number indicates better performance and therefore better client service. Note that this test was not run using Connection Keep-Alives, which is a new HTTP feature. Connection Keep-Alives are fully implemented in Microsoft Internet Information Server and IIS could go faster using them, but to keep the tests fair they were not used, because Keep-Alives have not been implemented in all browsers or servers.

[IIS1822D 5117 bytes ]

Figure 2. Connections per Second

Another way to look at this is Web transactions per second: how many transactions (accepting the request from the client and sending back the requested data) a server can drive per second. The results in Figure 2 clearly demonstrate that Microsoft Internet Information Server can handle as many as four times more transactions than the Netscape Communications Server on Windows NT Server, three times more transactions than Netscape on BSDI UNIX, and easily handle 50% more transactions than Novell's NLM Web server.

In real world terms, Microsoft Internet Information Server would be able to support four times the number of user requests than the Netscape Web server, thus allowing Web-site managers to support a greater number of users on similar hardware. It also means that users will experience a quicker response time. This advantage also manifests itself by allowing the site administrator to better prepare for peak bursts of user requests.

Back to table of contents

Average Response Time

Average response time measures the amount of time required to complete an operation once it is started. The WebStone benchmark measures both connection response time and transfer response time. Connection response time measures the time taken to establish a connection. Transfer response time measures the time to complete a data transfer once a connection has been established. User-perceived response time includes the sum of the connection plus transfer response times, plus any network latency due to WAN/LAN connections, routers, modems, and so forth.

[IIS1822E 5021 bytes ]

Figure 3. Average Response Time for Client Load

For this set of results, a lower number is better. Results in Figure 3 show that the measured average response time for the Microsoft Internet Information Server is significantly lower than that of the Netscape Communications Server. Although the Netscape server did not demonstrate any client errors, this graph indicates that it was forced to queue client requests, thereby imposing a frustrating delay on the end user to complete a request. This delay increased steadily as the number of client requests increased. Note that the average response time results for the 128 client load for the Netscape Communications Server for BSDI UNIX are not part of this graph because the server was unable to complete the test at this load.

Average response time on small networks

This graph is important because it also clearly demonstrates that the Netscape Communications Server, on both Windows NT and BSDI UNIX, imposes a delay on users even when the Web server is not heavily burdened. Looking at the average response time for the 16 client load tests, you can see that Netscape servers can take up to seven times longer to successfully respond to a client request. This clearly demonstrates that even for Web servers being used by a relatively small number of users, Microsoft Internet Information Server is clearly the better solution.

[IIS1822F 5124 bytes ]

Figure 4. Average Response Time for Each File

Keeping in mind that the typical Web page is made up of several files, usually a HTML file, and several GIF/JPEG or other graphics files, the average response time results in Figure 4 becomes even more important--because this is the response time for each file on the Web page. So, if a complex Web page with lots of text and graphics files is accessed on a Microsoft Internet Information Server, the page will appear extremely quickly, while a user accessing the same page from a Netscape server will be forced to stare at the hour glass for several seconds.

Back to table of contents

Application Programming Performance

Web server extensibility

Many Web sites are moving beyond the delivery of static HTML files. To deliver engaging and high-impact content, Web sites need to be dynamic and have the ability to generate pages that are filled with information targeted specifically at each user (as an example, see http://www.msn.com and the custom home page capabilities). In addition, customers and independent software vendors seek ways to improve site and content management, to use the Web as an extension to their own network and to deliver information technology founded on the Web as a platform. To deliver dynamic interaction and value-add extensions Web servers must offer a high-performance, easy-to-use programming model.

The method for delivering this new level of functionality is to extend the Web server itself programmatically. There are a variety of techniques available in today's Web servers: The most common approach is called the Common Gateway Interface, or CGI. CGI, which is supported by nearly all Web server implementations, is one of the simplest and most straightforward ways of extending the server. Netscape uses a proprietary method called NSAPI, that is available on only a limited number of Netscape's Web server implementations. (For example, Netscape Communications Server for BSDI UNIX did not include support for NSAPI at time of this publication.) Microsoft and other Web server vendors offer an open method called the Internet Server Applications Programming Interface, or ISAPI. The performance of a Web server extension API becomes an important factor in servicing high-impact content to the broadest possible user base.

ISAPI and CGI

The first data that needs to be examined is a comparison of the performance of a CGI application to an equivalent ISAPI application running on the Microsoft Internet Information Server. This test is configured to generate 100% CGI or ISAPI pages, with no static HTML pages. This will give a good baseline for the performance that users will see when using ISAPI applications rather than CGI applications on the Microsoft IIS platform.

The chart below compares WebStone results, at 64 clients, for 100% CGI or ISAPI, 0% HTML configuration. The charts show connections per second and throughput in megabits per second.

[IIS1822G 3506 bytes ]

[IIS1822H 3372 bytes ]

Figure 5. ISAPI vs. CGI

From these charts it is easy to see that in a direct comparison, ISAPI is close to five times faster than CGI at performing the same tasks on Microsoft Internet Information Servers.

For Web-site managers this means that by adapting CGI-based applications to the ISAPI interface, they will immediately see a decrease in the time that users have to wait to have their requests for data filled by the server, and the server will be capable of handling more requests. Moving CGI applications to ISAPI applications will also allow the Web-site manager to install more Web applications on a single server, thereby improving the impact of the site without adding additional server resources.

Back to table of contents

ISAPI and NSAPI

Once it was determined that ISAPI was considerably faster than CGI, the question remained: Is ISAPI faster than Netscape's proprietary Web server API (NSAPI), at the same tasks? To determine this Shiloh ran the WebStone tests, (with 64 clients, configured for 100% API, 0% HTML page generation), against the Microsoft Internet Information Server using an ISAPI port of the API test, and against the Netscape Communications Server for Windows NT running the NSAPI port of the API test. Note that this test could not be used to compare NSAPI on BSDI UNIX--Netscape does not currently support NSAPI on BSDI UNIX. Also, it should be noted that Novell's NetWare Web Server was not included in the CGI or API testing because Novell does not currently support this functionality. Novell's Web server does support a proprietary extension called Remote CGI (R-CGI) that allows a CGI application to be run on a server separate from the Web server (which means that for the Novell solution, a Web site would need one Web server and one CGI server to match the capabilities of a single Web server, either from Microsoft or the other Web vendors). Because this is not the standard way Web sites are configured, Novell's Web server was not included in this portion of the testing.

[IIS1822I 4467 bytes ]

[IIS1822J 4458 bytes ]

Figure 6. IIS ISAPI vs. Netscape ISAPI

The results, in both throughput per second and connections per second, clearly show that for performing the same tasks, ISAPI is considerably faster than NSAPI--almost 3 times as fast. Once again this indicates that users would receive a quicker response from their database query or other Web application when the application is written in ISAPI rather than using NSAPI. This also means that as more and more dynamic Web applications are added to a server, Microsoft Internet Information Server can handle more applications and users on a single server, without degrading the performance as perceived by the users.

Back to table of contents

Cross-platform Comparisons of Public Benchmarks

While the tests described above demonstrate the overwhelming performance characteristics of the Microsoft Internet Information Server, these evaluations all hold the tested hardware platform constant. The vendors of expensive UNIX systems sometimes claim performance as their unique value-add. To learn whether or not these claims can be sustained with a Web server as the application, a variety of publicly available performance information has been compiled and is displayed in the chart below. This chart uses the results of WebStone data running on Microsoft Internet Information Server on low-cost PC servers, contrasted to the competition. Vendors who have published WebStone 1.1 results include HP (HP 9000 Group) (see results at http://hpcc998.external.hp.com/csopress/96jan29a.html) and Spyglass (http://www.spyglass.com/products/server_results.html).

[IIS1822K 7266 bytes ]

Figure 7. Connections per Second

The results are telling: Windows NT Server and Microsoft Internet Information Server, running on a Hewlett-Packard®, Intel Pentium-powered Windows NT Server, performed two to four times faster than any of the other Windows NT or UNIX Web servers--including the "high performance" UNIX systems. In addition, Internet Information Server significantly outperformed Novell's NLM-based Web server.

For the Web site manager, the Information Systems or department manager implementing an intranet Web server, this means that they can use the lower-cost, easier-to-set-up-and-manage Windows NT Server-based Internet Information Server to provide a significantly faster response time to their customers. Combined with the performance data on ISAPI applications, it is clear that the best Web server, for standard HTML publishing or hosting Web-based applications, for both the Internet and the intranet, is Windows NT Server and Microsoft Internet Information Server.

Below are descriptions of the Web servers tested in the chart above:

[IIS1822B 12561 bytes ]

The Spyglass numbers used above were for test results for 60 clients, where as the HP and Microsoft tests were conducted for 64 clients. We believe that this difference in clients between tests does not diminish the comparative value.

It should be noted that the performance numbers for the Spyglass® Web server were not included in Figure 7 because the Spyglass performance numbers were obtained using Connection Keep-Alives, which significantly improves the results of benchmarks such as WebStone's. Spyglass' benchmark results with Keep-Alives were good (over 16 megabits per second throughput, and almost 300 connections/second at 60 clients), but they should be compared with results from benchmark tests where all servers use Keep-Alives. Microsoft Internet Information Server also includes support for Keep-Alives, and IIS could go faster using them, but in fairness, the benchmark tests were run without Keep-Alives because they are not implemented in all browsers and servers. Microsoft plans to publish benchmark results for IIS with Keep-Alives at a future date, when a model has been developed to use it appropriately for benchmarking.

Back to table of contents

NSTL Benchmarks

The National Software Testing Laboratory (NSTL) is a well known software laboratory that has long been involved in testing and benchmarking PC and LAN software. NSTL uses an internally developed Web server benchmarking tool to test various Web servers, including:

The Server for these tests

The NSTL benchmark tested the ability of the Web, or HTTP server to transfer HTML files. The results are expressed in transactions per minute score. A completed transaction in the NSTL benchmark consists of the client making a request for data from the server and then receiving the correct data. This test measures how many of these transactions a server can complete per minute.

The results from the NSTL reinforce the results that Haynes and Shiloh obtained using the WebStone tests. Microsoft Internet Information Server is the fastest Web server by a large margin. In the NSTL tests, IIS was over three times faster than the Windows NT-based servers from Netscape, O'Reilly, and Process software, and the BSDI UNIX version of the NCSA Web server. IIS was twice as fast as the Netscape server for BSDI UNIX, and bettered by one-and-a-half times the performance of the new Netware Web Server from Novell.

This performance translates into substantial benefits for Web site managers that install Microsoft Internet Information Servers:

[IIS1822L 7160 bytes ]

Figure 8. Transactions per Minute

Complete results for the NSTL tests will be published by NSTL in the near future.

Back to table of contents

Conclusion

The results of the WebStone 1.1 and the NSTL Web performance tests are very clear. When comparing Windows NT Web servers, Microsoft's Internet Information Server is up to four times faster than competitive Web server products, including the Web servers from Netscape, Process software, O'Rielly & Associates, and Novell. In comparison to more expensive UNIX solutions, Microsoft Internet Information Server is the clear performance leader.

While it is not possible to draw direct comparisons between the numbers from these tests and the specific number of "hits" or users that can be supported on a single Internet or intranet Web server; it is possible to draw the conclusion that Web sites that use Microsoft Internet Information Server and develop applications using the ISAPI will:

Therefore, better customer service will be provided than by sites that use other Web server products.

For sites presenting information that is generated dynamically or that wish to use the Web server as an application platform, this performance analysis demonstrates that the combination of Windows NT Server, Microsoft Internet Information Server, and ISAPI is the best solution for developing and deploying Web applications and Web server extensions.

It is also clear that the myth that the fastest Web servers are UNIX/RISC Web servers is a fallacy. Internet Information Server, on low-cost PC Server hardware, is substantially faster than the much higher priced Sun Netra, and HP 9000 Web server solutions. Organizations considering the installation of Web servers for virtually any application will find their requirements optimally addressed by the Microsoft Internet Information Server, the only Web server integrated with Windows NT Server.

Back to table of contents

End Note

The CGI and API tests are part of the WebStone tests: By means of CGI and API interfaces, WebStone requests a small program be run on the Web server to generate a random string of characters that is sent back to the client. Typically, a CGI program is an *.EXE file and an API program is a *.DLL file. The file is loaded when the Web server starts up under Windows NT. Based on the WebStone 1.1 specification, the operation of the test program (either CGI or API) is identical for all Web servers although the actual code will differ between ISAPI and NSAPI, and between a Windows NT platform and a UNIX platform.

Appendix: WebStone

The WebStone version 1.1 benchmark was used to generate the workload and measure the performance of the Web server. The WebStone benchmark was developed by Silicon Graphics Incorporated (SGI) as a tool for measuring Web server performance. SGI makes WebStone freely available in source form to whoever wants to use it. For more information regarding this benchmark, see: http://www.sgi.com/Products/WebFORCE/WebStone/.

Microsoft recognizes that the SGI WebStone performance tests may not be the best solution for comparing performance of Web servers, but in the absence of a single, widely accepted Web benchmarking standard, Microsoft used WebStone for the comparative analyses that customers demand.

The WebStone Tests were run by Shiloh Consulting and Haynes & Company at the facilities of Shiloh Consulting.

Back to table of contents

References

Please note that several of the links below point to servers that are not under Microsoft's control. Please read Microsoft's official statement regarding other servers.

Microsoft Internet Information Server

For more information and to download the Internet Information Server see:

http://www.microsoft.com/infoserv/
For Additional Performance Information see:

http://www.microsoft.com/infoserv/haynes1.htm
http://www.microsoft.com/infoserv/
For Internet Developer Information see:

http://www.microsoft.com/intdev/
For General Microsoft Information see:

http://www.microsoft.com

Haynes & Company and Shiloh Consulting Reports

For the Microsoft Internet Information Server Benchmark Report see:

http://www.microsoft.com/infoserv/haynes1.htm
For the Netscape NSAPI vs. CGI on SGI UNIX Server Report see:

http://www.haynes.com/haynes1/bench.html
For Shiloh Consulting see:

http://www.haynes.com/haynes1/shiloh.html

WebStone Material From SGI

http://www.sgi.com/Products/WebFORCE/WebStone/paper.html

http://www.sgi.com/Products/WebFORCE/WebStone/FAQ-webstone.html

http://www.sgi.com/Products/WebFORCE/WebStone/

WebStone Haynes & Co. Response From NCSA

http://www.ncsa.uiuc.edu/InformationServers/Performance/CGI/cgi-nsapi.html

HP 9000 WebStone Benchmark Data

http://hpcc998.external.hp.com/csopress/96jan29a.html

Spyglass WebStone Benchmark Data

http://www.spyglass.com/products/server_results.html

For More Information

For the latest information on Microsoft Internet Information Server, check out our World Wide Web site at http://www.microsoft.com/infoserv/.

For more information on ISAPI, visit the Internet developers section of the Microsoft Web server at http://www.microsoft/com/intdev/.

© 1996 Microsoft Corporation