The first thing they did was phone Lloyds (very sensible). Lloyds told them that there were no problems at their end and to clear cookies, add the site to the ActiveX trusted sites list (seriously, why is anyone using ActiveX these days?), etc. Still not working, so must be a problem with the customer's firewall.
So the customer phoned us. We pointed a browser at https://payments.corporate.lloydsbank.com/ (on an independent internet connection) and nothing happens - just sits there waiting. So clearly Lloyds are having problems.
Lets try some slightly lower level debugging:
# openssl s_client -connect payments.corporate.lloydsbank.com:443 -servername payments.corporate.lloydsbank.comAnd it just sits there... Eventually:
CONNECTED(00000003)And more sitting there doing nothing... Then eventually:
write:errno=104Oh dear... I should've got a certificate back from that but instead the web server dropped the connection. So a few of obvious problems to look into here:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
- It took a really long time for "CONNECTED" to appear...
- It took a really long time for anything to happen once we're connected...
- Finally, the web server failed to send us a certificate and negotiate an encrypted TLS session.
- There was a significant delay before the first packet (SYN) even appeared. So something else was going on before the connection was even attempted. A DNS problem was a good bet.
- The first packet (SYN) was resent about 3 times before the web server responded. This would also cause a significant delay in starting the connection.
dig payments.corporate.lloydsbank.comResulted in a long wait before failing - definitely a DNS problem then. Lets find the name servers responsible:
# dig payments.corporate.lloydsbank.com ns +traceLooking those up gives me:
<snip>
payments.corporate.lloydsbank.com. IN NS ns-lv6.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS ns-lv2.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS ns-lv7.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS ns-lv3.lloydsbanking.com.
ns-lv6.lloydsbanking.com. IN A 141.92.88.1So there are 11 name servers. And by trying to look up against each of those name servers I find that 9 of them are down. That means that about 82% of DNS requests are going to time out - at best things are going to be very slow while the customer's DNS server makes repeated DNS lookups and waits for each to time out; at worst it will fail to find a working DNS server and give up, rendering the website inaccessible.
ns-lv2.lloydsbanking.com. IN A 141.92.96.1
ns-lv7.lloydsbanking.com. IN A 195.171.195.169
ns-lv7.lloydsbanking.com. IN A 195.171.195.168
ns-lv7.lloydsbanking.com. IN A 195.171.195.167
ns-lv7.lloydsbanking.com. IN A 195.171.195.166
ns-lv7.lloydsbanking.com. IN A 195.171.195.165
ns-lv7.lloydsbanking.com. IN A 195.171.195.164
ns-lv7.lloydsbanking.com. IN A 195.171.195.163
ns-lv7.lloydsbanking.com. IN A 195.171.195.170
ns-lv3.lloydsbanking.com. IN A 141.92.104.1
To summarise:
- 9 out of 11 of Lloyds' DNS servers were down, resulting in intermittently very slow or even completely broken DNS lookups.
- If you managed to resolve the web server's IP address, it took a long time to accept the connection.
- If you managed to get a connection, the web server may fail to negotiate an encrypted TLS session with the client.
