Kev’s blog

This post assumes reasonable sys-admin Alfresco knowledge and assumes you are already familiar with setting up the Alfresco Repository in a cluster configuration and familiar configuring an Apache web-server instance. You should read the previous post first.

Since my last blog on this subject, there has been quite a bit of interest in load balancing Alfresco Share. This is good news but also means that customers and the community have found some issues that needed looking at – good, because more use of Alfresco testing means more stability for everyone when we fix the issues – that’s part of the fun of having a very active user community!

Three main points were raised:

1. A bug relating to a problem with dashboard template layouts not updating between web-tier nodes – which was quickly resolved for 3.4.8 and 4.0 see ALF-12318
2. A request for example load balancing config for Share itself as I only posted info on the forward proxy config for Share.
3. An issue raised in regards to a noticeable drop in performance when the cache config detailed in the previous post was added to each Share instance (which was required for load balancing multiple instances) - see this related blog post http://blog.alfrescian.com/?p=146 and ALF-12336

I’ll address all issues here and share some exciting performance related to item 3 also!

1. A bug that manifested when a user changed the template layout selection for a dashboard – for example from 2 to 3 column layout. The problem was that SpringSurf PageView objects were internally caching the Page object rather than just the PageId – easily and quickly fixed.

2. Config for load balancing Share is very similar to that for load balancing Alfresco.

Set up two tomcat instances containing “share.war” webapp with the “share-config-custom.xml” and “custom-slingshot-application-context.xml” config as detailed in the previously post. Remember the ports exposed by Tomcat need tweaking if you have those instances on the same physical machine – increment the HTTP and AJP and redirect ports in the tomcat/conf/server.xml config. Also ensure you have set the “jvmRoute” attributes to different values ready for the load balancing config, I used “tomcat3″ and “tomcat4″ as I now have a lot of servers running on a single box!

Create another Apache instance, I just did a copy and paste from the one I used to load balance the Alfresco cluster. Again bump the Apache listener port value if it exists on the same physical machine. Finally configure Apache “httpd.conf” to load balance against your Share web-tier instances:

# Reverse Proxy Settings (Share multi-instance load balancing)
ProxyRequests Off
ProxyPassReverse /share balancer://app
ProxyPass /share balancer://app stickysession=JSESSIONID|jsessionid nofailover=On
<Proxy balancer://app>
BalancerMember ajp://localhost:8019/share route=tomcat3
BalancerMember ajp://localhost:8024/share route=tomcat4
</Proxy>

Simply point your client browsers at your new Apache instance. If you have set up your Share instances to themselves use an Apache which is load balancing against an Alfresco cluster then you now have a full 2x Alfresco Cluster + Apache + 2x Alfresco Share + Apache set up!

This is great – BUT it leads onto point 3…

3. Scalability and fail-over capability has improved by having multiple Share instances – but individual performance per Share node is reduced. Now you may consider (as I did first) that this is expected, there is after all some additional work here going on – in the case of the Alfresco cluster it’s inter-node communication overhead, and in the case of the Share nodes it is reduced performance due to the caches that have been disabled. What’s apparent here is that if you just cluster Alfresco and keep to a single Share instance, we find that a single instance of Share can easily service a 4 node Alfresco cluster so in practice there is little need to load balance Share for performance reasons – but you certainly might want to for high availability reasons. Perhaps a price worth paying for the ability to remove and drop in additional nodes without your users knowing or having to update their URLs… But it would be nice if there wasn’t such a noticeable performance drop in Share.

The good news is that this has all changed in Alfresco 3.4.8/4.0.1 – in response to the community blog post and our drive to continually improve the performance of Alfresco, a new clustering technique has now been implemented for the web-tier.

For a load balanced environment, Alfresco Share now uses Hazelcast to provide multicast messaging between web-tier nodes. The end result of this is that all caches are now enabled again for each node, and we send very simple cache invalidation message when appropriate to all nodes. So the performance degradation is gone – each node is as fast a single Share instance.

The only changes required for each node are in “custom-slingshot-application-context.xml” – generally located in \tomcat \shared\classes\alfresco\web-extension and used to override the Spring application context beans for Share. There is an example “custom-slingshot-application-context.xml.sample” provided in the Alfresco distribution which now includes this config.
Enable this section on each Share tomcat instance to enable the Hazelcast cluster messaging:

   <!--
        Hazelcast distributed messaging configuration - Share web-tier cluster config (3.4.8 and 4.0.1)
        - see http://www.hazelcast.com/docs.jsp
        - and specifically http://www.hazelcast.com/docs/1.9.4/manual/single_html/#SpringIntegration
   -->
   <!-- Configure cluster to use either Multicast or direct TCP-IP messaging - multicast is default -->
   <!-- Optionally specify network interfaces - server machines likely to have more than one interface -->
   <!-- The messaging topic - the "name" is also used by the persister config below -->
   <hz:topic id="topic" instance-ref="webframework.cluster.slingshot" name="slingshot-topic"/>
   <hz:hazelcast id="webframework.cluster.slingshot">
      <hz:config>
         <hz:group name="slingshot" password="alfresco"/>
         <hz:network port="5801" port-auto-increment="true">
            <hz:join>
               <hz:multicast enabled="true"
                     multicast-group="224.2.2.5"
                     multicast-port="54327"/>
               <hz:tcp-ip enabled="false">
                  <hz:members></hz:members>
               </hz:tcp-ip>
            </hz:join>
            <hz:interfaces enabled="false">
               <hz:interface>192.168.1.*</hz:interface>
            </hz:interfaces>
         </hz:network>
      </hz:config>
   </hz:hazelcast>

   <bean id="webframework.slingshot.persister.remote" class="org.alfresco.web.site.ClusterAwarePathStoreObjectPersister" parent="webframework.sitedata.persister.abstract">
      <property name="store" ref="webframework.webapp.store.remote" />
      <property name="pathPrefix"><value>alfresco/site-data/${objectTypeIds}</value></property>
      <property name="hazelcastInstance" ref="webframework.cluster.slingshot" />
      <property name="hazelcastTopicName"><value>slingshot-topic</value></property>
   </bean>

   <bean id="webframework.factory.requestcontext.servlet" class="org.alfresco.web.site.ClusterAwareRequestContextFactory" parent="webframework.factory.base">
      <property name="linkBuilderFactory" ref="webframework.factory.linkbuilder.servlet" />
      <property name="extensibilityModuleHandler" ref="webscripts.extensibility.handler" />
      <property name="clusterObjectPersister" ref="webframework.slingshot.persister.remote" />
   </bean>

The config enables the Hazelcast Spring integration which starts the Hazelcast server, it is easily configurable and can use either multicast (the default and minimal effort) or TCP-IP direct if preferred. See http://www.hazelcast.com/docs.jsp for more info. For the default set up, identical config can be applied to each Share node and it will “just work”.

When you start Share you’ll see something like this:

INFO: /192.168.2.8:5801 [slingshot] Hazelcast 1.9.4.6 (20120105) starting at Address[192.168.2.8:5801]
19-Jan-2012 13:58:57 com.hazelcast.system
INFO: /192.168.2.8:5801 [slingshot] Copyright (C) 2008-2011 Hazelcast.com
19-Jan-2012 13:58:57 com.hazelcast.impl.LifecycleServiceImpl
INFO: /192.168.2.8:5801 [slingshot] Address[192.168.2.8:5801] is STARTING
19-Jan-2012 13:58:59 com.hazelcast.impl.MulticastJoiner
INFO: /192.168.2.8:5801 [slingshot]
Members [1] {
        Member [192.168.2.8:5801] this
}
19-Jan-2012 13:58:59 com.hazelcast.impl.management.ManagementCenterService
INFO: /192.168.2.8:5801 [slingshot] Hazelcast Management Center started at port 5901.
19-Jan-2012 13:58:59 com.hazelcast.impl.LifecycleServiceImpl
INFO: /192.168.2.8:5801 [slingshot] Address[192.168.2.8:5801] is STARTED

This means the config has driven the initialisation of Hazelcast successfully. That’s all there is to creating a Share instance in the cluster, if the config is present it will become a cluster node, if the config is not present (such as for a default install) then Hazelcast never starts . Once each node is started they will find each other automatically. Then once you users interact with Share, only when the following operations occur will cache invalidation messages will be sent from the affected node to the others in the cluster:

• an existing site/user dashboard layout is modified
• new site or user dashboard is created
• runtime app properties are changed (the Share theme currently)

This keeps chatter to a minimum and performance up!

This post assumes a reasonable sys-admin level of knowledge of Alfresco and assumes you are already familiar with setting up the Alfresco Repository in a cluster configuration and also familiar configuring an Apache web-server instance.

For high availability and performance reasons, clustering the Alfresco Repository is a commonly used setup. But it is also easy to use Apache for both load balancing an Alfresco Share web-tier against an Alfresco cluster and also to cluster (i.e. web farm) the Alfresco Share web-tier itself – or both those combined together. This post examines the configuration required for those setups using Alfresco 3.4/4.0.

These are the three setups we will be considering. Note that all examples assume you have already successfully setup a working Alfresco 3.4/4.0 repository cluster. It also assumes you are deploying the Share web-tier(s) to separate TomCat instance, not the same one as the Alfresco repository instances.

NOTE: all the example config below mentions “localhost” – as for testing purposes I set this all up on a single machine, changing the various TomCat ports as I go – but in reality you would probably be using difference machine instances (virtual or otherwise) for each, so care must be taken when setting up the config!

For setup 1. Configuring Share to point to an Apache reverse proxy that is setup to load balance to a multi-node Alfresco repository cluster.

You should be familiar with “share-config-custom.xml” – generally located in \tomcat \shared\classes\alfresco\web-extension and used to override the common Share startup parameters, including the remote endpoint locations. There is an example “share-config-custom.xml.sample” provided in the Alfresco distribution as a starting point. So add this section:

   <config evaluator="string-compare" condition="Remote">
      <remote>
         <endpoint>
            <id>alfresco-noauth</id>
            <name>Alfresco - unauthenticated access</name>
            <description>Access to Alfresco Repository WebScripts that do not require authentication</description>
            <connector-id>alfresco</connector-id>
            <endpoint-url>http://localhost:8089/alfresco/s</endpoint-url>
            <identity>none</identity>
         </endpoint>

         <endpoint>
            <id>alfresco</id>
            <name>Alfresco - user access</name>
            <description>Access to Alfresco Repository WebScripts that require user authentication</description>
            <connector-id>alfresco</connector-id>
            <endpoint-url>http://localhost:8089/alfresco/s</endpoint-url>
            <identity>user</identity>
         </endpoint>

         <endpoint>
            <id>alfresco-feed</id>
            <name>Alfresco Feed</name>
            <description>Alfresco Feed - supports basic HTTP authentication via the EndPointProxyServlet</description>
            <connector-id>http</connector-id>
            <endpoint-url>http://localhost:8089/alfresco/s</endpoint-url>
            <basic-auth>true</basic-auth>
            <identity>user</identity>
         </endpoint>
      </remote>
   </config>

The endpoint-url elements are the important bits – set them to your Apache location, in my example this is “localhost:8089”.
And in your Apache httpd.conf file, enable the various proxy/balancer modules (these modules are used in both examples – so you might not actually need them all, but there is no harm in enabling them):

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_http_module modules/mod_proxy_http.so

And then add this section:

# Reverse Proxy Settings (cluster load balancing)
ProxyRequests Off
#NOTE: sticksession only required if non-default Alfresco auth stack is used i.e. NTLM2 or similar
#stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPass /alfresco balancer://app
ProxyPassReverse /alfresco balancer://app
<Proxy balancer://app>
  #Add your Alfresco cluster nodes here
  BalancerMember ajp://localhost:8009/alfresco route=tomcat1
  BalancerMember ajp://localhost:8014/alfresco route=tomcat2
</Proxy>

The various cluster nodes should be added into the Proxy element as above, then in each Alfresco node instance TomCat server.xml, ensure the AJP connector is enabled and that the port matches the BalancerMember setting above, also ensure the Engine element jvmRoute attribute is set to match i.e using the above example:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

and

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">

Note that sticky sessions are not required by default when using the standard Alfresco authentication stack as the communication between Share and Alfresco is completely stateless and does not require a web Session. This means that your Apache instance can load balance the requests, even multiple Ajax requests from a single page instance, across multiple Alfresco cluster nodes to give the best performance. However, the side effect of this is that Lucene results will potentially differ in a cluster and Lucene is used to retrieve even simple things like Share document list and various picker results in 3.4 and below. The good news is that from Alfresco 4.0 and onwards, Lucene is no longer used for list or picker results by default (canned DB queries are used instead), so sticky sessions are not required in 4.0.

For setup 2. We need to configure an Apache forward proxy to point to multiple Share web-tier instances, also each user browser must be configured to point to the Apache proxy. Load balancing can of course be used, although the simple Apache config below doesn’t show that. This time however sticky sessions are required between the browser and the Share instance as a small amount of Session state is managed for each user in Share.

Apache httpd.conf (not a load balanced example but you get the idea)

# Forward Proxy Settings
ProxyRequests On
ProxyVia On
<Proxy *>
  Order Allow,Deny
  Allow from all
</Proxy>

You may be familiar with “custom-slingshot-application-context.xml” – generally located in \tomcat \shared\classes\alfresco\web-extension and used to override the Spring application context beans for Share. There is an example “custom-slingshot-application-context.xml.sample” provided in the Alfresco distribution as a starting point.
So add this section  to each Share tomcat instance to disable the appropriate web-tier caches (this example actually comes from the currently shipped custom-slingshot-application-context.xml.sample, which is where the helpful comments come from):

   <!-- Web-tier cluster configuration -->
   <!-- Enable this section if you are clustering or load balancing the share.war i.e. multiple web-tiers behind a proxy -->
   <!-- If you have a single web-tier running against an Alfresco cluster via a reverse proxy you don't need this -->
   <bean id="webframework.slingshot.persister.remote" class="org.springframework.extensions.surf.persister.PathStoreObjectPersister" parent="webframework.sitedata.persister.abstract">
      <property name="store" ref="webframework.webapp.store.remote" />
      <property name="pathPrefix"><value>alfresco/site-data/${objectTypeIds}</value></property>
      <property name="noncachableObjectTypes">
         <set>
            <value>page</value>
            <value>component</value>
         </set>
      </property>
   </bean>

For setup 3. Use a combination of the above two setups! So modify the “custom-slingshot-application-context.xml“ and “share-config-custom.xml” for each Share TomCat instance to configure the forward and reverse proxy Apache instances (you will need two Apache instances for this) and to disable the web-tier caches.

With Alfresco 3.3, the Share content collaboration platform has never been easier to develop for! With the integration of SpringSurf in Alfresco 3.3, plus a number of improved debugging features, the edit->test->debug development cycle for Share pages and components is more rapid than ever. Once set-up, it can take literally seconds to see the results of changes to any WebScripts, templates, JavaScript or CSS files. This post is a collection of tips direct from Alfresco Engineering on how to set-up your environment, tweak the default Share configuration, debug scripts and enable the fastest possible development cycle.

Get the Tools You Need

Your favorite developers text editor. Obviously. But that’s pretty much all you need to work with Alfresco WebScripts, FreeMarker templates and JavaScript text files – you don’t need anything heavier, but something like Eclipse or IntelliJ will do nicely if you prefer.

Firefox plus the Firebug extension or Google Chrome – because of the nifty HTML, CSS inspection and JavaScript debugging features. Most Alfresco devs prefer the Firefox+Firebug combination as the features are currently stronger than the dev tools built into Chrome, and it gives more accurate and descriptive JavaScript errors – but Chrome is catching up with every release and it’s certainly noticably faster at processing JavaScript and general rendering.

Setting up a Folder Structure

See this previous post on how to set-up your folder structure for creating Share pages, components and WebScripts. It also discusses using the SpringSurf resource servlet to easily access web assets such as images, CSS and client-side JavaScript in your templates and components from within the same folder structure.

During Development Deploy Files and Folders – not a JAR!

Create a quick shell or ant script to copy your working files and folder structure to your tomcat Share classes folder. You can then work on your assets easily and quickly deploy them all to a running Alfresco server. The previous post describes how to package them up for run-time deployment as a single JAR – but that isn’t needed during development, and in fact isn’t really desirable – having the files and folders expanded out ensures quicker deployment and ensures the various config options discussed below will work correctly.

share-config-custom.xml

Share is configured by various Spring config files and Alfresco format config files. These are easy enough to override. There’s really only one file you need to worry about for most situations. The default config override file that Share is already set-up to look for is called share-config-custom.xml and should be placed in your $TOMCAT_HOME$/shared/classes/alfresco/web-extension folder. Here you can override various configuration settings and several of the most useful ones for development and debugging will be discussed here. If you do want to override Spring beans, then Share will look for a file called custom-slingshot-application-context.xml and it should be placed in the same folder. Generally unless you are modifying Surf internals you won’t need to override the Spring beans.

A default installation of Alfresco will include a .sample version of these files in the above location, they can also be found in the source tree: HEAD/root/projects/slingshot/config/alfresco/web-extension

Debugging client-side JavaScript

Here’s an example of a share-config-custom.xml file with the correct config to turn on client-side JavaScript debugging. You may be wondering why you need to do this, after all surely it’s just a matter of turning on FireBug and getting on with it? But for performance reasons, Share will include minimized versions of its client-side JavaScript files during page rendering – it will even combine some of the files – which don’t make for useful debugging. Turning on the client-debug flag forces Share to load the original uncompressed source with all comments and formatting intact, perfect for debugging into. It also ensures the various YUI library files are uncompressed – if you are feeling brave enough to poke around in them.

   <alfresco-config>

      <!-- Global config section -->
      <config replace="true">
         <flags>
            <!-- Developer debugging setting - DEBUG mode for client scripts in the browser -->
            <client-debug>true</client-debug>

            <!-- LOGGING can be toggled at runtime when in DEBUG mode (Ctrl, Ctrl, Shift, Shift).
                 This flag automatically activates logging on page load. -->
            <client-debug-autologging>false</client-debug-autologging>
         </flags>
      </config>

   </alfresco-config>

Restarting the server is required if you make changes to this file. Force a refresh of Share in your browser and you will then be able to use FireBug or the Chrome debugger tool to set breakpoints and debug the client-side JavaScript of Share components.

You’ll notice the client-debug-autologging flag in the example above. There is a logging feature in Share that allows a developer to make use of log4j style debugging methods in client-side JavaScript. It’s not a commonly used feature as it’s pretty low-level, but if you can’t solve your issue with breakpoint style debugging then logging may be useful. The flag is used to automatically active the logging output window in Share rather than having to toggle it with a funky key combination.

To log from within your client-side JavaScript:

if (Alfresco.logger.isDebugEnabled())
   Alfresco.logger.debug("some string value");

The log4j info, warn, error and fatal levels can be used if required.

SpringSurf Development Mode

SpringSurf has a number of autowire configuration modes. By default Share is configured in the “production” mode – this ensures that web-tier JavaScript and FreeMarker templates are compiled and cached for maximum performance. For development, it is much more useful to have JavaScript and templates not cached. As mentioned above this allows simple scripts to quickly copy over new versions of files and see the results immediately in the application – no restarts required.

This is an example of the config required for share-config-custom.xml:

   <alfresco-config>

       <config evaluator="string-compare" condition="WebFramework">
           <web-framework>
               <!-- Autowire Runtime Settings -->
               <autowire>
                   <!-- Pick the mode: development, preview, production -->
                   <mode>development</mode>
               </autowire>
           </web-framework>
       </config>

   </alfresco-config>

Web-Tier JavaScript debugging

Share SpringSurf components are simply a special case of an Alfresco WebScript – with a few additional rules and restrictions on what objects are available and what presentation templates can output. The web-tier JavaScript controller associated with a WebScript can be debugged via the Rhino debugger window. Once “development” mode is activated as mentioned above, the script debugger will be able to break on JavaScript functions and breakpoints. To turn on the debugger, use the browser to navigate to /<yourserver>/share/service/index – this will display the WebScripts index page. Click the Alfresco JavaScript Debugger link and then Enable. A Java GUI window will appear, this is the Rhino debugger interface. The easiest way to get into the web-tier JavaScript is to select Debug->Break On Function Enter from the menus. The next WebScript that is executed in the app will break in the debugger. Once the appropriate source files are loaded breakpoints can be added and the Break On Function Enter can be disabled to reduce the noise during debugging.

Web-Tier Javascript Logging

Logging from within WebScript code can be another useful debugging aid. On both the Repository and Web-Tier, the “logger” root scoped JavaScript object is available for simple logging tasks. The output is directed to the App-Server console window.

logger object API:

boolean logger.isDebugEnabled() - returns true/false to indicate if debug logging output via log4j is active,
this can be used to ensure performance isn't degraded when console logging is inactive.

void logger.log(string message) - output a log message to the console

To turn on logging for your application, modify the log4j.properties file and set the following class to debug:

log4j.logger.org.alfresco.repo.jscript.ScriptLogger=debug

Note that the same logging class name is used by the Alfresco repository and SpringSurf – it is maintained for backward compatibility.

Refreshing WebScripts

On a production deployed instance of Share or any instance where all the various debug settings noted above are not applied, most caching and refresh issues can be quickly resolved without restarting the app-server. The Refresh WebScripts option in the Share WebScript index page can be used to clear the caches for all components and compiled scripts. It also means that additional WebScripts can be deployed or removed at runtime without restart. Use the browser to navigate to /<yourserver>/share/service/index – this will display the WebScripts index page. Click the Refresh Webscripts button to clear the caches and refresh the list of available WebScripts.

Separate Alfresco and Share Server Instances

Since Alfresco 3.0, the Share client is a stand-alone web-application and the share.war can be deployed on a separate TomCat instance to the Alfresco repository, on another server entirely if required. This is recommended for development and production. For development, it enables the Share application to be restarted quickly (a few seconds on a modern laptop or server) without interfering with the main Alfresco repository instance. For production, it allows proxies and clustering to be implemented.

The configuration to do this is simple, but first it requires the set-up of a new TomCat instance, with appropriate tweaks as recommended for an Alfresco server such as; JVM memory settings, server URIEncoding set to UTF-8, web-extension folder etc. The easiest way to do this is to take a copy of the Alfresco installed TomCat instance and remove the alfresco.war. If you are running the new TomCat instance on the same machine as the Alfresco repository instance then modify the port settings in tomcat/conf/server.xml to ensure Share can be started at the same time as Alfresco. Developers at Alfresco generally use port 8080 for Alfresco and port 8081 for Share.

By default the Share web-application will attempt to communicate with the Alfresco server via HTTP on port 8080. There should be no additional configuration required if you use those settings. However if your Alfresco instance is not running on this port then the following example of configuration for share-config-custom.xml can be used to modify the endpoint settings, change the endpoint-url values as appropriate:

   <!-- Overriding endpoints to reference a remote Alfresco server -->
   <config evaluator="string-compare" condition="Remote">
      <remote>
         <endpoint>
            <id>alfresco-noauth</id>
            <name>Alfresco - unauthenticated access</name>
            <description>Access to Alfresco Repository WebScripts that do not require authentication</description>
            <connector-id>alfresco</connector-id>
            <endpoint-url>http://localhost:8080/alfresco/s</endpoint-url>
            <identity>none</identity>
         </endpoint>

         <endpoint>
            <id>alfresco</id>
            <name>Alfresco - user access</name>
            <description>Access to Alfresco Repository WebScripts that require user authentication</description>
            <connector-id>alfresco</connector-id>
            <endpoint-url>http://localhost:8080/alfresco/s</endpoint-url>
            <identity>user</identity>
         </endpoint>

         <endpoint>
            <id>alfresco-feed</id>
            <name>Alfresco Feed</name>
            <description>Alfresco Feed - supports basic HTTP authentication via the EndPointProxyServlet</description>
            <connector-id>http</connector-id>
            <endpoint-url>http://localhost:8080/alfresco/s</endpoint-url>
            <basic-auth>true</basic-auth>
            <identity>user</identity>
         </endpoint>
      </remote>
   </config>

Conclusion

Hopefully this list of tips should help speed up development and improve your experience when customising and working with Alfresco Share components. In the next post we will explore the architecture of SpringSurf components, templates and page rendering.