TIL Thursday: Troubleshooting DNS Problems with the CDN

Content delivery networks (CDNs) are geographically distributed servers that specialize in delivering static content quickly. You can use them on your SuiteCommerce Advanced to optimize the performance of your site by using them to cache things like images, search results and SSP files.

Two crucial parts of getting the Akamai service (the CDN SCA sites use) to work is configuring the settings properly in NetSuite and with your DNS provider. Incorrect configuration accounts for a significant amount of the issues we receive so I sat down with Mark Sweeting, a senior web performance engineer here at NetSuite, in order to get some troubleshooting advice.

NOTE: modifying the configurations of the CDN and DNS is only recommended for experienced users.

Basic Checks

Unfortunately, CDN problems don't typically present symptoms that immediately point to a CDN or DNS problem. For example, the most common symptom is simply slowness, and slowness can be caused by a number of things. Rather dramatically, you're entire site being unavailable could point to a DNS issue.

Of course, this is ambiguous and there are some checks you can perform. The simplest is to use a browser extension for Chrome called Akamai debug headers. You may recall from a previous article on caching that we recommended this plugin. In short, what it does is as add a number of HTTP headers to requests along with some custom Pragma headers, so that when you inspect a HTTP request in the developer tools, useful additional information is given.

Once you have the extension added, visit your site's homepage and inspect your homepage in the Network tab. Under the Response Headers section you'll see a number of new responses.

To know if the page is cached and your CDN is working fine, the X-Check-Cacheable property should say "YES".

Check Your Setup

Next, it's important to see if you've got the basic setup correct.

Start by identifying your site's CNAME. The CNAME is used as an alias for another domain, so it works like a redirect. To find it, go to Setup > SuiteCommerce Advanced > Domains. The CNAME (ALIAS) column shows the auto-generated alias used by NetSuite to identify your website to the CDN. It'll look something like "www.yourbusiness.com.hosting.netsuite.com". Copy this to your clipboard.

Then you need to update the DNS settings to use this CNAME alias. In a separate tab, log in to the control panel of your DNS provider and locate the DNS settings for the SCA site domain. Remove any A records that exist for the domain and replace them with a new CNAME record, using the alias you copied.

NOTE: it's important that when you edit or update this setting you include the "www." prefix in your record if it was included in the CNAME generated by NetSuite. So, for example, www.yourbusiness.com and yourbusiness.com are considered two different domains, so you must make sure you're editing the right setting. Also note that email is typically sent to a non-www prefixed domain, so be careful to check you're updating the appropriate record. If you see any MX records (mail exchanger records) you should be careful to leave them in place, as changes to these could break your email.

Finally, back in NetSuite, enable the CDN by selecting the checkbox in the CDN column next to your SCA site.

DNS changes can take some time to propogate. Usually it takes up to a few hours (but sometimes up to a day) before your changes take effect.

If after some time it's still not operational, then take a look at some of the following steps.

Common DNS Problems

Broadly speaking, most CDN issues we encounter are caused by one of the following:

  1. Your DNS CNAME is configured correctly, but the CDN is not enabled
  2. Your DNS CNAME is not configured correctly
  3. Your DNS changes haven't had time to propogate yet

Before I talk about the first two, I want to introduce a useful tool. Available as standard on Mac and Linux-based operating systems, the dig command queries DNS servers and returns useful information about a domain. If you're running Windows, it is not available by default but can be installed separately.

So, if I were to run dig on an example site, I get the following response (which I've trimmed):

$ dig www.scaexample.com

;; ANSWER SECTION:
www.scaexample.com. 14398 IN    CNAME   www.scaexample.com.hosting.netsuite.com.
www.scaexample.com.hosting.netsuite.com.    298 IN CNAME website-cdn.na1.netsuite.com.
website-cdn.na1.netsuite.com. 59 IN CNAME   website-cdn.na1.netsuite.com.mdc.edgesuite.net.
website-cdn.na1.netsuite.com.mdc.edgesuite.net. 317 IN CNAME a1280.b.akamai.net.
a1280.b.akamai.net. 8   IN  A   67.132.183.51
a1280.b.akamai.net. 8   IN  A   67.132.183.49

There are two things to note here: firstly, you can see the CNAME being translated, and, secondly, you can see the hits to Akamai. Thus, we know that the CNAME is set up correctly and the CDN is enabled.

Problem 1: CNAME Configured Correctly, CDN Checkbox Not Checked

;; ANSWER SECTION:
www.scaexample.com. 14099 IN    CNAME   www.scaexample.com.hosting.netsuite.com.
www.scaexample.com.hosting.netsuite.com.    299 IN CNAME shopping.na1.netsuite.com.
shopping.na1.netsuite.com. 2962 IN  A   64.89.45.13

So we can see the CNAME is translating but its translating to the standard NetSuite shopping servers, not the Akamai ones. If this looks like the answer section for your site, you probably don't have the CDN checkbox checked.

Problem 2: CNAME Configured Incorrectly

;; ANSWER SECTION:
www.scaexample.com. 2997    IN  A   123.123.123.123

If you get this answer then you haven't configured the settings with your DNS provider correctly; log in to the control panel for it and follow the steps above.

Problem 3: DNS Propagation and Caching

If you're sure your CNAME is configured correctly with your DNS provider and you've enabled CDN, but you're still not seeing the right answer from dig then it could be to do with propagation or caching. The first solution to this problem is simply to give it time. You can't force propagation to be any quicker and sometimes it can take up to a day before all servers have been updated.

We can, however, use dig to give you some information about when you can expect the DNS cache to refresh. Let's look again at the dig trace from our properly configured server:

;; ANSWER SECTION:
www.scaexample.com. 3389  IN  CNAME   www.scaexample.com.hosting.netsuite.com.
www.scaexample.com.hosting.netsuite.com.    298 IN CNAME website-cdn.na1.netsuite.com.
website-cdn.na1.netsuite.com. 44    IN  CNAME   website-cdn.na1.netsuite.com.mdc.edgesuite.net.
website-cdn.na1.netsuite.com.mdc.edgesuite.net. 21551 IN CNAME a1168.b.akamai.net.
a1168.b.akamai.net. 20  IN  A   23.62.237.89
a1168.b.akamai.net. 20  IN  A   23.62.237.97

You'll see after each domain there is a number; this is the TTL (time to live), which is the remaining time for which the lookup is considered valid and cacheable. The first CNAME answer is 3389 seconds, which is about 56 minutes; thus you'll have to wait 56 minutes before this is looked up again. If you look at the bottom two, the values for the Akamai servers are very short: only 20 seconds. This is because Akamai uses DNS to dynamically route requests from the users to edge nodes close to them. In practice, therefore, it's usually only the ones relating to NetSuite servers that cause issues.

So you're best bet at the moment is to periodically run dig until those numbers reach 0 and the cache is reset. Once it has, check again and see if the CDN is working.

Finally, if you're able to flush your DNS cache then it could help. But if a DNS server on your network or ISP is holding a cached DNS answer, it will be difficult to work around it. Again, waiting for cached records to expire may be your only option. In some circumstances you can workarounds this with cache invalidation — I've included links to documentation about this below.

Further Reading

At SuiteWorld 2016, Mark Sweeting will be hosting a breakout session on optimizing SCA sites for performance (DEV1578).