Tech Forum

From Joel Dehlin's blog:
[INDENT]We apologize for the slowness of lds.org the last two weekends. We’ve recently moved to a new infrastructure and haven’t yet figured out the right configuration for caching and garbage collection.

If there are any Websphere/Vignette experts out there, make yourself known in the LDS.ORG forum over at our tech web site!

Thanks for your patience.
[/INDENT]I have WebSphere knowledge, but not Vignette. Mainly wanted to echo his call for experts in this forum.

Paul

I'm no WAS expert, but this article may be of interest to the tech staff:

http://www.alphaworks.ibm.com/tech/cach ... P=AWRSSJAV

Has the problem been resolved? The site, lds.org, seems to be running fine but I wanted to know if there were still open issues on application tuning.

bhofmann wrote:Has the problem been resolved? The site, lds.org, seems to be running fine but I wanted to know if there were still open issues on application tuning.

I don't know if the issue has been resolved but I know that we have had a team working hard on it.

Tom

tomw wrote:I don't know if the issue has been resolved but I know that we have had a team working hard on it.

Tom

Thanks. I didn't know if we should still be searching for possible solutions. Many of us are not Vignette experts but we know quite a bit about tuning java applications and application servers in general.

For the most part the problem has been solved. We did a couple of things to fix it. Although I am not part of the middleware team that was involved with the physical implementation of tweaking it, I was on most of the phone bridges discussing the tuning of the boxes.

The major problem was that the caching appliance (NetCache) was not configured just right. It had been configured, but there were a few key pages that weren't being cached and would do a full table scan and regenerate the menu structure for the site.

The reason that this wasn't picked up the first round was because we had turned on another site (ldsnewsroom) at the same time and although its usage shouldn't have been a factor it was. The load that we threw at the box the first time wasn't enough with the load of newsroom and lds.org.

The other configurations that we made were to add more RAM and another processor to each node of the cluster. Although this seems like a no-brainer, adding more RAM and procs can actually make the problem worse. After adding everything we went through a another few rounds of rigorous JVM and server tuning for the added hardware. We made adjustments to the heap and extended memory as well as other areas.

Once it was all said and done we had thrown 6 times the load at it that we were previously seeing when lds.org was struggling and it came through with shinning colors.

I want to say thanks on behalf of ICS for everybody's ideas, thoughts and prayers while we sorted through this.

Excellent! I know times like this can be frustrating. At work, we recently went through a large web site launch and had some JVM performance issues. We did a lot of heap and permgen tuning and session management adjustments to solve our problem. It sounds similar to what you had to do. Great job in getting it solved.