Digital Edition

Benchmark Bust-Up:The Middleware Company Responds
Benchmark Bust-Up:The Middleware Company Responds

(November 8, 2002) - As controversy swirled around the validity of the .NET vs J2EE benchmarking results posted this week by The Middleware Company, JDJ editor-in-chief Alan Williamson went straight to TMC for their side of the story.

JDJ: Did TMC approach Microsoft with respect to the production of this report? If so, did TMC approach BEA, IBM, Oracle, or SUN for similar participation in the production of the report?

TMC: Yes, we approached Microsoft. Our original intention for this benchmark was to put forth a more realistic comparison of J2EE and .NET to replace the original unfair PetStore application that Microsoft published. We challenged Microsoft to see if they were interested in a new comparison involving some recodes of the J2EE implementation and the .NET implementation. They were interested, and helped us by providing a testing lab and covering some of our costs.

We did not approach the J2EE vendors. The vendors' general position on tests such as these are that they do not want to get involved, because they want benchmarks to be performed by committees rather than by independent entities. If we do a round 2 benchmark, we will invite the J2EE vendors, so that they have the opportunity to get involved.

JDJ: You are quoted to having using the Sun's recommended best practices (blueprints) for the optimizations of the PetStore application. Why were the best practices as founded and published by TMC not employed?

TMC:The TMC team did work hard and made the J2EE system run 17x faster than the original PetStore application. However, after considering the community's feedback, we have concluded the system was not fully optimized for performance. There are important and necessary optimizations that were not performed. They were not performed primarily due to artificial time and money constraints on the project. However, we realize now that if we are to do any comparison of J2EE and .NET, then both must be shown at their finest.

We do believe the optimizations we did not have time to make would have made a material impact on performance. Because of this, if you are considering making a purchase decision involving J2EE and .NET, we encourage you to take a "wait and see" approach and look towards a second benchmark, which we are currently considering performing. Going forward, we are considering doing a second benchmark that takes the community's completely valid critique into account. Please stay tuned to for developments in this area.

JDJ: Did you enter into any contract with Microsoft for this? Were you legally obligated to publish your findings however they turned out?

TMC: Yes. The agreement was that regardless of who the victor was, the results would be published.

JDJ: Many J2EE vendors have commented that TMC didn't properly represent the J2EE community. When TMC engineers found the results to be going against J2EE, why wasn't outside assistance from the J2EE vendors sought?

TMC: The position of many J2EE vendors is that they do not want to get involved with independent benchmarks. They do not believe in the viability of PetStore as a useful benchmarkable application, and are spending their time and energy on committee-based benchmarks. This is a completely valid position for the vendors to take, and we respect that position.

JDJ: Sun has continually said that the PetStore application should never be used for performance testing, not even between J2EE app servers. Yet the PetStore was still used as the basis for this benchmark. What benefit did TMC see from continuing to use the PetStore application?

TMC: We used PetStore for two reasons:

1) We would have liked to port a real committee-based benchmark such as ECPerf to .NET, but that code is protected by Sun and we currently do not have permission to port it to .NET and perform a comparison.

2) Microsoft originally published a bogus report comparing a .NET PetStore with a J2EE PetStore, and showed .NET to be 10x faster. This was particularly embarrassing since, after all, they had used Sun's example. We wanted to debunk that comparison and do one more realistic -- for example, one that did not use stored procedures, which was in Microsoft's original comparison. So PetStore was the natural choice.

JDJ: We believe you are already thinking of revisiting the report with a new set of tests. Will Microsoft be involved? Have you invited the participation of BEA, IBM or any other J2EE vendor?

TMC: Yes, if we do a round 2, then we will invite the J2EE vendors to participate. Please stay tuned to for developments in this area.

JDJ: Speaking now on a personal level, what message would you like to give to the Java community who may be a little stunned at your findings?

TMC: Our advice is to not use this report when making purchase decisions, but rather, to either look towards committee-based benchmarks, or towards a round 2 PetStore bakeoff which we are considering doing right now.

About Alan Williamson
Alan Williamson is widely recognized as an early expert on Cloud Computing, he is Co-Founder of aw2.0 Ltd, a software company specializing in deploying software solutions within Cloud networks. Alan is a Sun Java Champion and creator of OpenBlueDragon (an open source Java CFML runtime engine). With many books, articles and speaking engagements under his belt, Alan likes to talk passionately about what can be done TODAY and not get caught up in the marketing hype of TOMORROW. Follow his blog, or e-mail him at cloud(at)

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1

The only good to come from this fiasco is to show that when somebody asks how does A compare to B, the only correct answer is to do the comparison yourself, because you can't trust the so called experts to do it for you.

Do you go and trade your Chevey for a Honda when Honda wins a formulae 1 race? And then again, trade your Honda in for a Ferrarri when they win the next one? No. Why? Because your car does exactly what you need it to do at a cost much,much lower than the formulae 1 cars, or even the show-room Ferrarris.

Do you trade in your computers every 2 months because there's something faster on the market? No. Why? Because your computers are fast enough for what you use them for.

I'm a java programmer, but I don't give a rat's butt what the "final" results are. I'm developing very cost effective Java solutions on Linux machines for my clients. I've always been able to meet the performance targets required.

Remember, not everyone is Michael Schumiatcher nor are they developing the fastest Chess program on the planet. Millions of companies simply want to get a quality, cost-effective solution to a business problem.

When a client has Windows servers and is already using Visual Basic, I work with them to develop VB-based solutions. When they don't, I steer them towards a less expensive, but just as productive Java/Linux solution. If they insist on going the Microsoft route, so be it. All I, and we as IT professionals, can do, is provide enough information for them to make the best choice for thier company.

I've written only 1 small VB app in the last 2 years. Businesses out in the real world understand that IT is suppose to be about solving business problems, not about holy wars.

Fr what it's worth,

TMC: "Our advice is to not use this report when making purchase decisions..."

Why did they bother going to all that effort when the result is useless? Many people have now expended lots of time discussing and debunking this "benchmark" - time that could have been put to more productive use.

TMC has no credibility left in this area and should be overlooked as a source of advice regarding .NET versus J2EE.

Question: did TMC approachthe J2EE vendors for participation in the production of the report?

November 8 TMC Answers: We did not approach the J2EE vendors.

This information should have been in the report.
This information should have been in your FAQ.

Hey guys,
The interview speaks by it self.
That starts as: "...we challenged MS"
and ends as: "... due to artificial time and money constraints".
All these years these guys are representing the "Java-bus" we take freely from a "Java-station".
Some day they "challenged" the "MS-bus" ... and left out of gas.
I find the message very clear and in the right way:

The evil empire is at it again since Java and Linux is eating up its market share it is sweating a lot can you smell stinks. Now with all its cash that it gets from MSOffice and desktop OS it is fighting for its life to have monopoly. VB has been a total disaster so now thier copy cat C# and .Not framework is a flop too. Well this should be easy just boycott MS products from your work and home and this company will disappear. We have switched to linux last year our expenses and Java dev time has been cut to half the time ..say bye bye to M$.

The Borg is definitely up to its old tricks, fresh from its buyout of the american justice system.

But with a twist this time around - such FUD can only mean one thing - extreme desperation.

Here are some tasty rebuttal factoids:

- embedded linux has the same market share, but higher growth rate, than M$ embedded.

How long before M$ embedded is toast in embedded market?? Embedded is >>>>>>> desktop.

Linux has higher growth than M$ in server market. How is M$ going to stop Linux?

If M$ cannot stop Linux in server and embedded, how long before the penguin eats desktop?

- Mindcraft played similar games a while back. How come no more mindcraft potshots at Linux?

All mindcraft did is open the door wider for Linux, and, I rather suspect, that the bounce of this little maneuver will strengthen Java too.

It is starting to look like Big Bad Bill$ is his own worst enemy.

How about another license fee hike for Office????

Make My Day....

A couple of points:

1. Who really trusts benchmarks? Anyone with a bias can find or generate benchmarks to support their claims. They are, perhaps, food for thought and further investigation cannot be fully trusted when used as a marketing tool. Is Microsoft the only vendor that "buys" benchmarks? Hardly... What it comes down to is that I don't trust ANYONE's benchmarks but my own :)

2. This J2EE / .NET feud is just silly. Realistically, each is strong in places where the other is lacking... there is no "perfect" choice here - only a choice that is perfectly suited to your environment, budget, skills, and ideology. We dropped Java in favor of .NET this past year because for us it made the most sense: licensing and maintenance for our Java tools was astronomical, our development targets Windows exclusively, and we really missed the rich UI we were used to in native Windows apps. Is that the choice everyone should make? Absolutely not! If you are happy with what you use, *be* happy... but don't disparage other choices.

It just seems that this turns into a case of people wanting to bash anything Microsoft does, even when they do something good!

I think the next benchmark for both .Net and J2EE should be run on Linux boxex -- I suspect J2EE will be infinitely faster... ;^)

I haven't see such a fast backpedal since my bicycle brakes went out going down "Suicide" hill when I was 10.

This benchmarking comparison was advertised as a fair comparison between best practice implementations of .NET and J2EE. Now TMC themselves are stating that the results of this benchmark are not to be trusted?

This from a company that is supposed to be experts at implementing and mentoring enterprise J2EE implementations?

The irony is so deep here that I can't even conceptualize an effective metaphor to compare it to. Wow.

Java community - let's not lose sight of the real issue here - Microsoft bought and paid for this benchmark. TMC is going to pay for this charade with a massive loss of credibility. Just don't forget that MS has not stopped in it's efforts to undermine the Java industry, and although we may be able to see through their marketing hype, there are plenty of uneducated decision makers out there that are reacting to marketing fluff such as this.

Excerpt 1: "Microsoft originally published a bogus report comparing a .NET PetStore with a J2EE PetStore, and showed .NET to be 10x faster."

Excerpt 2: "The TMC team did work hard and made the J2EE system run 17x faster than the original PetStore application."

The question: why TMC J2EE solution is still slower?

Subscribe to the World's Most Powerful Newsletters


With the proliferation of both SQL and NoSQL databases, organizations can now target specific fit-fo...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ag...
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savin...
Your job is mostly boring. Many of the IT operations tasks you perform on a day-to-day basis are rep...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO an...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As au...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018,...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news an...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held Novemb...
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning te...
A traditional way of software development efforts reimbursing is pay by the hour, which in case of r...
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with e...
CI/CD is conceptually straightforward, yet often technically intricate to implement since it require...