One of our most popular sessions at SuiteWorld 2018 was one led by Diego Cardozo and Mark Sweeting on troubleshooting ecommerce performance. In their talk, the duo gave nine practical steps developers can take to troubleshoot performance problems with their site.
In this post, I want to revisit their talk and summarize the key points. Some of them we have already talked about before, and more detail will be available in the performance checklist, but I want to reiterate them and add a little detail.
1. Content Delivery Network (CDN)
Right off the bat, I'm going to link you to two previous blog posts, as this is something we covered a little while ago, see:
For the uninitiated, a CDN replicates and stores (caches) files stored on an origin server (eg NetSuite) and makes them available around the world. The prime benefit of a CDN is that it makes server responses a lot faster as they are:
- More likely to be geographically closer to the user than the origin server (lower latency)
- Less likely to be as busy as the origin servers (load balanced)
- Do not need to generate the files or data they are serving, they merely send the latest copies (super responsive)
It's the final point that will typically constitute 'caching problems' — a common, non-performance related, complaint is that the user is being sent the wrong version of a file. In other words, the CDN will have a cached version of a file but it will be out of date.
However, from a performance point of view, the most common issue our performance engineers see is that the CDN is not working at all, which means that we're not providing the fastest response to visitors.
The first step is to check whether you've turned on the CDN. Yes, one of the most common problems is that developers forget to turn on the CDN after development has finished.
When you're developing a site, it is common to deactivate caching because you are making frequent updates to the site and want to avoid old files being served (or having to constantly invalidate the cache). It's easy, therefore, to forget to turn it on when the site goes live and begins trading.
For instructions on how to set up CDN caching, see Enable CDN Caching.
If it is enabled but still appears to be malfunctioning, then you can test it using the CLI tools of
dig (Unix) or
nslookup (Windows) like this:
dig IN A www.example.com
nslookup -class=IN -querytype=A www.example.com
When you run one of those commands, you should see your site's domain resolve to the CDN provider, eg Cloudfront:
www.example.com. 9 IN CNAME www.example.com.hosting.netsuite.com.
www.example......netsuite.com. 299 IN CNAME abcdefghijklmn.cloudfront.net.
abcdefghijklmn.cloudfront.net. 59 IN A 184.108.40.206
abcdefghijklmn.cloudfront.net. 59 IN A 220.127.116.11
abcdefghijklmn.cloudfront.net. 59 IN A 18.104.22.168
abcdefghijklmn.cloudfront.net. 59 IN A 22.214.171.124
abcdefghijklmn.cloudfront.net. 59 IN A 126.96.36.199
abcdefghijklmn.cloudfront.net. 59 IN A 188.8.131.52
If, however, you do not then you'll be able to solve particular problems.
For example, if the CNAME is OK but the CDN in Setup > SuiteCommerce Advanced > Domains is not checked, you'll see it resolve to NetSuite servers like this:
www.example.com. 600 IN CNAME www.example.com.hosting.netsuite.com.
www.example...ting.netsuite.com. 300 IN CNAME 9999999-origin.shop.netsuite.com.
9999999-origin.shop.netsuite.com. 300 IN CNAME shop.source.netsuite.com.
shop.source.netsuite.com. 66 IN CNAME shopping.netsuite.com.
shopping.netsuite.com. 286 IN A 184.108.40.206
If you haven't created a CNAME record then the A record will point straight to the NetSuite data center:
www.example.com. 600 IN A 220.127.116.11
If your site is running on Akamai, then there's a plugin you can use in Chrome to debug the headers that are returned from their servers. This is something we've looked at before and can read about in my conversation with Raphael Tucat.
2. The SEO Page Generator and TTFB
As we know, the purpose of the SEO page generator is to create full HTML documents from our source code so that search engines can crawl our sites. We've talked before about how you can use it to debug code and troubleshoot SEO problems, but performance issues are a different matter — these typically come in the form of slowness in time to first byte (TTFB).
TTFB is the delay between the browser receiving content after requesting it. When a user visits a page (and requests content), the process goes something like this:
- Look up domains using DNS
- Make a TCP connection
- Negotiate security (ie TLS/SSL)
- Send request for content
- Wait for the time to first byte
- Begin downloading content
You can see the TTFB by inspecting the request your browser makes in its Network tab, eg:
There are also many, many documents online about diagnosing slow network performance using browser developer tools, eg, this page from Google.
So, while there are generic things you can try do to get this number lower, we mention the page generator because there are specific things to look for when debugging with it.
If after you've attached the debug and cache-busting URL parameters and you're examining the log, take a look at the sub-requests. There's a few things you can look for:
- Sub-requests that take 1000s of milliseconds rather than 100s
- Redirects (they return
302 status codes)
- Calls to services that you don't use
Furthermore, you may experience performance issues if your sub-requests fail outright (eg return a 502 error). If those calls enable other bits of functionality and are then not present, they could affect the performance of your site.
A handy tip: if your site is not yet running Prerender, turning it on for the purposes of debugging (by adding
&seoprerender=T to URL) could prove useful. The debug feedback from Prerender might give more detailed clues to your site's poor performance.
3. Content Optimization
This is a big area, and a good place to start is with a site called WebPagetest.
After submitting your site's URL to them, they will assess your site on a number of performance factors. Each factor is then graded and details provided. It makes for interesting reading.
For example, the waterfall view can be used to highlight whether you have a lot of requests, or if there are specific responses that are blocking the site from loading.
However, we mention it in the section for content optimization because it is useful to see the breakdown of your site's content. Take the following example:
In the above example, we can see that a big culprit are its images. Optimizing your images for better performance is something Mark and I have discussed before, and it's something you should absolutely be doing. That blog post has a lot of detail for reducing the file size of your images, but don't forget that NetSuite has a built-in service for resizing them. Make sure you read the documentation on setting this up and you annotate your code appropriately.
4. Page Rendering
While we're using WebPagetest, take a look at its rendering timeline functionality (similar functionality is also available in Chrome). What this does is take a screenshot of your site during the loading process, at intervals you can define (eg every 100 milliseconds). This is important because this affects the perceived performance of your site, ie, how fast your site looks to visitors.
Diego and Mark say that the three places you should look first are:
- Custom fonts
- Slow, blocking responses (ie they prevent parallel downloads)
Custom fonts: do you really need them? They can have a seriously detrimental effect on your site's performance. Consider going without them.
The second two are often related. It's not uncommon for us to investigate a site and to find that several tracking tags (for analytics or ads) have been implemented. These can also have a massive impact on your site's performance. You can test their impact by using the Block request domain feature in Chrome, or by simply running your site with a sophisticated ad blocker (many these days block analytics tags too).
5. Item API
By now, we've looked at the CDN, page generator and your site's content as potential problems. If you're still experiencing slowness, then there's more areas to look at, and these are more specific to NetSuite. The first one is the item API.
Results from the item API are produced from a Solr query on the application server, which indicates which items are to be returned. With that information, a database query is run on the database server, which returns data for those items. You can actually test this yourself by attaching the
ssdebug=T to the end of an API call.
The crucial thing is the
sqltime value, which in my case indicates that my search for 225 items took 1040 milliseconds. You would need to worry if that number was much larger than that.
The most common cause of slowness in the item API is unnecessary fields. This is something I talked to Pablo Zignani about a while ago. The essential guidance is that some fields are bigger than others, and will take longer to generate data for. A classic example is stock information: the
isinstock is generally quite light, but returning all stock information for an item is very intensive. You need to take a look at your field sets and how you're using them and ask yourself whether you could remove unnecessary fields from them, even if this means having to show less information in certain places.
Take another example: you want to show stock information, but consider whether it is important this information is made available on product list pages. What about custom fields? Sometimes, the performance team have found that data about each item's related items have also been included, which, on a PLP, is completely unnecessary. Therefore, you can make a modification to the fieldset used by your PLP so it's optimized like this:
Diego says he hasn't seen one SCA site that doesn't have at least one unnecessary field in their configuration, so take a look — quite often a lot of sites still use of default configuration, which is set up to be appropriate for a lot of sites (rather than specific aimed at a particular site).
6. Environment Files
These are the files that typically load first when your site is called; you'll know them as shopping.user.environment.ssp and shopping.environment.ssp.
The user environement file contains data that is only appropriate to the current user. As such, it is not cached. However, the standard environment file is cached and it can be used to bootstrap information if it needs to be loaded early and available globally (which we looked at in our previous webinar on data).
However, if you're a cheeky developer, or have employed one, then it's entirely possible that there's something in these files that is slowing your site down. It's pretty easy for developers to lazily 'dump' code in these files because of their primacy and, in the case of the user environment file, are not cached. However, putting non-essential code in these files can be particularly bad because they both process synchronously, blocking the rest of the application until they're finished.
Some shopping.user.environment.ssp tips:
- Remove non-user-specific information
- Check if there's any unnecessary 'keep fresh' code (ie data that has been placed here because it is never cached)
- As this is a global file, you'll probably notice the slowness across the entire web store
As for shopping.environment.ssp:
- Check for unnecessary bootstrapped data (ie, is it required — and is it required across the entire site?)
- Consider replacing the bootstrapped data with a service
- As this is where the site's categories, CMS and configuration data is called, check to see if any of these are the culprits
Honestly, quite often the biggest problem is category data: it really needs to be optimized!
On the subject of categories, it's worth keeping in mind that they can be quite intensive to generate and so you should approach their use with performance in mind.
The team's three golden rules are:
- Use commerce categories — some people have shoe-horned in our old category functionality 🙁
- Don't create category trees that go deeper than three levels
- Try to assign a product to only one category
Generally speaking, you want to keep your categories trim. There seems to be this mentality among some people who think that they need to add products to every category possible, which ends up with very deep and overfull categories that really don't accomplish much segmentation. Remember, categories are there to aid your shoppers to find what they want: swamping them with repetitive products isn't going to help!
8. Scriptable Cart
We've looked at scriptable cart before and we know it's something that a lot of people use on their website. If you don't know, this is effectively a script that usually runs on both front- and backend sales and is a way of listening to events and either canceling them or spinning off additional actions. It's pretty common for third-party integrations.
Depending on what your script is set up to do, that's where you may end up seeing performance problem. For example, if your scriptable cart listens for changes to order line items, or when new ones are added, then you'll likely see issues where adding items to the cart is slow. We've also seen a number of performance issues around the checkout wizard because of scriptable cart.
You can perform some testing by going to Customization > Scripting > Scripted Records > Sales Order and look at all the scripts that run on it.
After picking a suspect, you can go to its deployment record and toggle the All Roles and All Employees checkboxes: if you check All Roles and uncheck All Employees, it'll run in the web store; if you uncheck All Roles and check All Employees, it'll run only in the backend. Note, though, that this is a bit hacky and isn't a long-term solution: you need to figure out why the script is taking so long!
9. User Events
If there are issues with how long it takes to place an order then you may need to examine your user event scripts. There's an option called Execute in Commerce Context on a script's deployment record and it determines whether it will run on web stores — if you toggle this off and things improve, then this is likely where your problems lie.
There's also a feature you can enable called Asynchronous afterSubmit Sales Order Processing they may help you if your code listens for the afterSubmit event. When this feature is enabled, afterSubmit user events are triggered asynchronously when a sales order is created during web store checkout, meaning that they will run in the background.
When an afterSubmit user event runs in asynchronous mode, the next page is loaded during the period of time the afterSubmit user event is still processing. You may find some joy if you switch to asynchronous event handling.
The steps can be summarized as follows:
- Check the content delivery network (CDN)
- Check the SEO page generator
- Optimize your content (use WebPage Test)
- Optimize page rendering (fonts, third parties, blocking requests)
- Optimize the item API responses
- Check environment files
- Review your category structure for speed
- Review scripts that impact cart timings (scriptable cart)
- Review user events
Practical advice on improving your web store's performance is available in detailed or checklist format, via our performance page.
If you suspect an SCA SuiteScript file of being slow and want to try and narrow down which call or function, take a look at something I wrote a little while ago about using console timers to track how long it takes for things to run.