Tuesday, 18 September 2018

Chasing wild geese

So, a quick one this morning.  One of our customers has been having problems accessing Lloyds Bank's corporate payments gateway.

The first thing they did was phone Lloyds (very sensible).  Lloyds told them that there were no problems at their end and to clear cookies, add the site to the ActiveX trusted sites list (seriously, why is anyone using ActiveX these days?), etc.  Still not working, so must be a problem with the customer's firewall.

So the customer phoned us.  We pointed a browser at https://payments.corporate.lloydsbank.com/ (on an independent internet connection) and nothing happens - just sits there waiting.  So clearly Lloyds are having problems.

Lets try some slightly lower level debugging:
# openssl s_client -connect payments.corporate.lloydsbank.com:443 -servername payments.corporate.lloydsbank.com
And it just sits there...  Eventually:
And more sitting there doing nothing...  Then eventually:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 0 bytes
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
Oh dear...  I should've got a certificate back from that but instead the web server dropped the connection.  So a few of obvious problems to look into here:
  1. It took a really long time for "CONNECTED" to appear...
  2. It took a really long time for anything to happen once we're connected...
  3. Finally, the web server failed to send us a certificate and negotiate an encrypted TLS session.
Firing up tcpdump, I found that:
  1. There was a significant delay before the first packet (SYN) even appeared.  So something else was going on before the connection was even attempted.  A DNS problem was a good bet.
  2. The first packet (SYN) was resent about 3 times before the web server responded.  This would also cause a significant delay in starting the connection.
So, investigating a potential DNS problem:
dig payments.corporate.lloydsbank.com
Resulted in a long wait before failing - definitely a DNS problem then.  Lets find the name servers responsible:
# dig payments.corporate.lloydsbank.com ns +trace

payments.corporate.lloydsbank.com. IN NS  ns-lv6.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv2.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv7.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv3.lloydsbanking.com.
Looking those up gives me:
ns-lv6.lloydsbanking.com. IN    A
ns-lv2.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv7.lloydsbanking.com. IN    A
ns-lv3.lloydsbanking.com. IN    A
So there are 11 name servers.  And by trying to look up against each of those name servers I find that 9 of them are down.  That means that about 82% of DNS requests are going to time out - at best things are going to be very slow while the customer's DNS server makes repeated DNS lookups and waits for each to time out; at worst it will fail to find a working DNS server and give up, rendering the website inaccessible.

To summarise:
  • 9 out of 11 of Lloyds' DNS servers were down, resulting in intermittently very slow or even completely broken DNS lookups.
  • If you managed to resolve the web server's IP address, it took a long time to accept the connection.
  • If you managed to get a connection, the web server may fail to negotiate an encrypted TLS session with the client.
With multiple Lloyds Bank servers having serious problems, I wouldn't mind betting that they are being attacked.  But why didn't Lloyds' own support people know / admit that there were problems on their end rather than sending the customer on a wild goose chase - it only took us 15 minutes to diagnose a problem that Lloyds' own people must have already known about.

Tuesday, 31 July 2018

Why do they not listen?

I don't usually talk much about our customers, but sometimes things happen which truly beggar belief.

For many years we have been contracted by a consortium of schools who were geographically close and originally wanted to be able to share a single connection for financial reasons.  This is quite a common arrangement.  Due to the shifting landscape of internet provision, costs and politics, the arrangement came to an end some time ago, which is fine - projects don't last forever, customers come and go and the project ended amicably, with all of the schools involved being pretty happy with us.  In fact, they all ended up taking out independent contracts with us for one thing or another after the project ended anyway.

Just one of the schools has been somewhat... "odd" at times though (I will refer to them simply as "School" to retain their anonymity), and despite our best efforts it has gradually caused problems for them.  Their ICT support is outsourced, which can be both good or bad - some of the companies that provide outsourced ICT services are pretty good, but some of them seem to have a "rip everything out" attitude and insist on unnecessarily replacing a school's equipment on day one instead of spending some time looking to see what is working well and what isn't and only replacing the stuff that isn't working.  This generally seems to be because they want to use systems that they are already used to rather than learning something new.  There's some merit in trying to standardise on systems you know, but obviously leads to a lot of disruption and expense for the school, so in my view is not a great way of doing things.

Anyway, the story probably starts in 2013.

Summer 2013

We had been providing the connectivity between the schools for some years by this point.  Because of limitations of the technology which was available at the time when it was installed, the schools had unreliable, but redundant, interconnects.  These weren't installed by us, but we were contracted to provide and maintain systems to use those unreliable interconnects to provide reliable connectivity.  We were also contracted to provide online safety systems (web filtering, etc.) to the whole consortium.

School: We've just changed our ICT provider and the new provider has decided to replace the existing online safety system with a third party system.
Us: That's fine, but the connections you use for internet access aren't reliable enough to use independently and the equipment you're proposing to remove is used to provide reliable internet access over those unreliable connections. The third party equipment that you want to install is incompatible with the protocols used by the existing equipment, which means you will also need to replace the equipment at the far end of the connections.

Obviously we prefer not to lose a customer, but if they want to switch to another provider then that's fine and we try to guide them and minimise the disruption as best we can.

The ICT provider ordered the third party system, plugged it in, discovered that it didn't work with the interconnects (as we had told them) and ended up backtracking on the whole thing.  So they left our online safety system in place and in fact I think their ICT provider probably decided it was ok in the end - at least they made no further moves to replace it.

Summer 2015

School changed ICT provider again.  This didn't really have any impact on us.

Summer 2016

By now, all of the schools in the consortium, except for School had installed their own internet connections and the interconnects were just being used for a small amount of local traffic and as redundancy in case of failure of one of the internet connections.  The consortium decided that the project had run its course and announced that it would be dismantled by summer 2017.

There was also some discussion about retiring the infrastructure early:

Consortium: We should simplify things by removing all of the equipment that is managing the connections immediately.
Us: We can't do that because that equipment is still need to provide the connection to School.
Consortium: No problem, we'll leave it as-is until summer 2017.

Spring 2017

School: We intend to continue using the existing interconnects after the summer.
Us: It isn't economic to do so since you're now footing the whole bill instead of it being shared by the whole consortium.  The existing equipment is also very old so liable to fail soon.  Replace everything with a new connection, it'll cost less than continuing with the existing equipment.  We can do this for you or you can get a third party in, we don't mind either way.

Summer 2017:

Us: As agreed last year, the existing interconnects will now be shut down.  School will need to migrate over to their new connection.

School: No alternative connection has been procured, there's no time left to get one now, we need to keep using the existing connections.
Us: We already said this was uneconomic, but as a good will gesture we'll take some of the hit ourselves and knock 50% off the cost.  But this is a one year only deal - we will not support this next year because the equipment is well past its end of life.  Also, as the equipment is very old, we recommend you follow our original recommendation and replace the interconnect ASAP since it might fail at any point.  If any of the hardware fails, we won't fix it.

Spring 2018:

Us: Just a reminder, you need to replace the interconnects ASAP.

Summer 2018:

School: We've just changed our ICT provider (again) and the new provider has decided to replace the existing online safety system with a third party system.
Us: That's fine.  As you already know, the connection you are using for your internet access is going out of service this summer, we presume you've procured a replacement?
School: No we haven't, we intend to continue using the existing interconnects.
Us: But those interconnects aren't reliable enough to use independently and the equipment you're proposing to remove is used to provide reliable internet access over those unreliable connections. The third party equipment that you want to install is incompatible with the protocols used by the existing equipment, which means you will also need to replace the equipment at the far end of the connections.
Us: In fact, this is exactly what we said in the summer of 2013, then again in the summer of 2016, then again in spring 2017, and in summer 2017, and in spring 2018.
Us: Also, we already told you a year ago that we weren't going to support any of this equipment which manages those interconnects any more as it is far too old.
School: We don't need that equipment, we're just going to use a single (15 year old, unreliable) connection in isolation.
Us: Errm, that will be really unreliable, here are some statistics from our monitoring data to show just how unreliable it will be.

Panic ensues at School.

School: Why didn't you tell us that we would need a new connection!  It's now far too late to procure one in time.  You must extend our contract for free.
Us: Umm, no.
Us: We've made every effort to recommend the most economic and reliable way forward and have been ignored at every step.
Us: Last year we dug you out of a hole you'd made for yourselves and even gave you a big cost reduction out of our good will.  You have repaid us by continuing to ignore our recommendations, blaming us for the mess you've got yourself in, paying your last invoice months late and cancelling your contract with us.
Us: You have now demanded that we dig you out of a hole again out of the goodness of our hearts at extremely short notice.
Us: Here are a selection of get-out-of-jail cards from our standard price list, which we are happy for you to buy from us at the standard price, go pick one.

And apparently this is all our fault...  At least now that the contract with them has ended we won't have to deal with any of the fallout from this mess.  Seems like a classic case of "I think we've heard enough from the experts" to me :)

Thursday, 5 April 2018

Thoughts on the gender pay gap

I've already managed to upset someone on Facebook because I apparently said that women are worth less than men (I didn't).  I don't regard myself as a feminist, but rather an equalist.  Discrimination of all forms is bad - women shouldn't be discriminated against based on their gender, but similarly men shouldn't be discriminated against in order to give women an advantage.

Discrimination is as old as time itself but I don't think you fix the problem by just changing which group you discriminate against, any more than you prevent war by changing which group of innocent people you're blowing up.

The British government is now requiring all businesses with over 250 employees to publish figures summarising the gender pay gap.  There are a few bits of information required, but essentially this boils down to a simple average difference in pay between men and women.

The press is then using the figures to bash the companies with the biggest pay gap.  No consideration is being given to what types of work are being done since that isn't in the information that companies are expected to publish.  i.e. this is not a like-for-like comparison.

There are a few potential reasons why women may be earning less than men:
  1. Maybe there are less women qualified and applying for the higher paid jobs.
  2. Maybe employers are refusing to employ women in the higher paid jobs.
  3. Maybe employers are employing women in the higher paid jobs but are paying them less than the men who are doing the same work.
All of these points are, of course, a problem.  The two last points are directly under the employer's control and any employer undertaking in this kind of discrimination deserves to have the book thrown at them - I firmly believe that equally qualified men and women should be given the same opportunities as each other and the same pay for doing the same work.

However, it is fairly unclear to me how the first point can be regarded as a specific employers' fault.  Since the information that is being made available doesn't do anything to differentiate between these possibilities, it seems completely unfair to vilify an employer based solely on this data.

So we have STEM companies and the construction industry with a fairly big pay gap simply because it's very difficult to recruit women to do the higher paid jobs in these fields.  I'm sure that in some cases there is discrimination going on, but you can't determine that from the data being used by the press to attack employers.

To demonstrate the issue, lets take a simple fictional employer - they aren't discriminating and they have the following breakdown of employees:

Total Men Women Salary
Engineers 200 180 20 £50,000.00
Admin staff 50 15 35 £20,000.00
Total 250 195 55

The mean pay is £47,692.31 for men and £30,909.09 for women, yielding a pay gap of £16,783.22, even though everyone doing the same job is paid the same.

So, how can the employer fix their pay gap?  Since we're already assuming there is no discrimination going on, we can look at it rationally with maths:

1. Increase the proportion of female engineers
It's very unclear to me how a single employer can increase the proportion of applicants who are female.  In order to do this the proportion of women training in engineering needs to be increased, starting with school kids.  There is some scope for the industry as a whole working to promote engineering to women, but it takes years and a single employer can't do a lot on their own.

At work, when we were last recruiting, we didn't end up hiring a man because we're horrible people who support the patriarchy; we ended up hiring a man because no women applied for the job.  We would usually want to pick the best person for the job, regardless of their gender.

So with far less women in the engineering job market than men, the immediate options are:
  • Increase the women's benefits and decrease the men's.  The women will be "overpaid" with respect to other employers and want to work for you whilst the men will be "underpaid" and get a job elsewhere.
  • Recruit underqualified women to make up the numbers, since there aren't enough qualified women applying.  Recruiting people who aren't qualified to do the job sounds like a bad idea for the business.
  • Restrict the number of applications from men.
All of these options discriminate against men based solely on their gender and are therefore pretty unethical (and probably illegal).

2. Increase the proportion of male administrative staff
So given that we probably can't do a lot to recruit more female engineers, we could tweak the balance elsewhere in the business.  The average women's pay is being dragged down by the fact that a disproportionate number of women are employed in the lower paid administrative roles.  Only 10% of the engineers are women, but 70% of the administrative staff are women.  If the employer reduces the proportion of women in the admin roles down to 10%, that will eliminate the pay gap.

This problem is pretty much the opposite of (1) - the same options apply, but this time the employer must discriminate against women.  Again, doesn't strike me as a good plan.

3. Increase the women's pay or decrease the men's pay
So far, the employer has paid men and women the same amount for the same work, which has led to a big pay gap simply because of the job role demographics.

Adding about 55% to the womens' pay in both job roles eliminates the pay gap.  We're now paying female engineers £77,000 and female administrative staff £30,800.  Of course, the employer may well not be able to afford these kinds of expenses, especially when the men find out that they are earning far less than the women and take their employer to court for sexual discrimination.

Similarly, reducing the men's pay by 65% across the board achieves a similar result - the employer gets sued into the ground for sexual discrimination, and if they survive that, with the men now being paid far below the market rate, they all leave for greener pastures.

4. Pay the engineers and administrative staff the same
Another option is to decrease the pay the engineers receive and increase the pay the administrative staff receive.  By paying everyone £44,000 irrespective of what job they do, the pay gap is eliminated whilst keeping the total wage bill the same.

Unfortunately, the company's engineers are now underpaid relative to the market rate, so they will leave and it will be impossible to recruit replacements.

The entire industry could follow suit, but this would lead to a long term shortage in engineers - it costs tens of thousands of pounds and several years to become an engineer, and how many people would do that if their pay is the same as someone who hadn't spent that time and money on training?

I am a firm believer that equally qualified and experienced women and men are equally valuable and should be paid the same.  For some jobs it is easier than others to ensure that this is happening - where there are fixed non-negotiable pay scales things are obviously clearer than jobs where employees are expected to negotiate their salaries.  In the latter case, people who are poor negotiators are obviously not going to do as well as good negotiators.  I don't know if there's a gender bias when it comes to negotiating skill, but any protection should surely extend far beyond the work place since poor negotiators also lose out when negotiating other things, such as buying a car, etc.

There are also big differences in demographics that need to be accounted for: is a 35 year old who graduated from university at 21 and has been in relevant employment for the whole time (giving them 14 years industry experience) as valuable as someone else who took 10 years off between 23 and 33 to bring up their children (so they have only 4 years industry experience)?  These people could be either men or women, but in the current society the majority of child carers are women and it doesn't seem right to completely discount that when comparing pay.

From taking a rational look at the data being collected, it seems clear to me that it is tackling the wrong thing.  Whilst I'm sure that discrimination is happening, the data being published cannot be used to determine who is discriminating.  Indeed, in many cases reducing the pay gap seems to actually require the employer to discriminate, so the whole thing seems very counter productive to me.

The things that need to be tackled are:
  • Ensuring that men and women are given the same opportunities.
  • Ensuring that men and women receive similar pay for similar work.
  • Ensuring that neither men and women are put off from taking any opportunities that are open to them.
Lets work on those points instead of attacking employers for things that they can't do much about.

Monday, 30 October 2017

ESP8266 CurrentCost Wifi Gateway

I have two CurrentCost power monitors - a CC128 EnviR panel in the living room which monitors our entire electricity usage, and an old white first generation panel in the office which monitors the office's electricity usage.

The CurrentCost panels output real time data in XML format every 6 seconds through a TTL serial connection presented as an 8P8C connector on the bottom, and I used to have a WRT54GL wireless access point connected to that, which sent the data to the server.  The WRT54GL was retired some time ago, so I wanted another way of recording the power usage data.  The ESP8266 seemed like a good choice to provide a bridge to my wifi network and I've chosen to use an ESP-12F board.  I could build a small enough device on stripboard to sit in the cavity under the base of the panel and connect to the CurrentCost's TTL serial output.

The 8P8C connector also includes regulated and unregulated power connections, but the 3v PSU is only rated for 80mA which is nowhere near enough to power the ESP8266.  I didn't want to have a second PSU, so opted to use a single USB power supply to power everything.  I've used an LM1117T LDO regulator to drop the supply voltage down to 3.3v, and connected this to both the ESP8266 and the unregulated supply rail of the CurrentCost.  There's a schottky diode between the regulator and the CurrentCost - this isn't strictly necessary, but will protect against someone plugging the CurrentCost PSU in at the same time as the wifi bridge.

GPIO0 and GPIO2 are pulled up by 10KΩ resistors, GPIO15 is pulled down by a 10KΩ resistor - you can probably get away with omitting the resistors and connecting these directly to Vcc / 0v respectively, although using a resistor to pull up GPIO0 allows the ESP8266 to be programmed in circuit by temporarily pulling it down to 0v.

CH_PD and reset are pulled up all the time by tieing them to the Vcc rail.

The connections to the CurrentCost's 8P8C connector are:
  • Pin 1 - Vcc / unregulated power
  • Pin 4 - Ground / 0v
  • Pin 8 - TTL serial data
Component-side view on the left, flipped for a track-side view on the right
Since the ESP-12F has solder pads with a 2mm spacing and the stripboard has 0.1" hole spacing, the ESP has to stand off the stripboard a little.  I've taken the opportunity to use the space between the ESP-12F and the board for the pullup / pulldown resistors.  I've shown the ESP-12F as a translucent block on the stripboard layout above so you can see the components under it.

The power connector is a surface mount mini-USB socket - obviously that's incompatible with stripboard, so I have just tacked on a couple of wires to connect it to the circuit and soldered its mounting lugs down to the track-side.  I find it useful to hot-glue the wires onto the board for mechanical support to avoid breaking off the socket's pins.
Component-side view

Track-side view
Although the CurrentCost outputs XML, I'm not bothering to parse that in the ESP8266.  Helpfully, each XML sentence is terminated with a newline, so it's easy to use Serial.readStringUntil('\n') to grab the whole sentence and post it to the web server.  Data posted over HTTP needs to be URL encoded and there doesn't seem to be a standard library to do this, so I ended up writing a quick and dirty encoder.  A PHP script on the web server parses the XML and feeds the data into RRDTool, and I'm using Cacti to display graphs of the data.

I wanted my graphs to display a total number of kilowatt hours used.  Unfortunately Cacti doesn't have a generic function to do this.  It can do bandwidth summation for network usage (i.e. displaying a "total megabytes" on a "bytes per second" graph), which is almost what I need.  Using that on the "watts" data gives a total number of watt seconds (i.e. joules) used, but no way to divide this by 3600 to produce watt hours.  In the end I added a COMPUTE data source to the RRD file, which records watts divided by 3600, and Cacti can then use that to calculate the total watt hours of electricity consumed.  Since I'm abusing the bandwidth summation functions, this unfortunately adds a "B" (bytes) unit to the total displayed, but I can live with that.

The code, schematic and layout are available from my Subversion repository.  The circuit works for both the CurrentCost EnviR (also known as the CC128) and the first generation CurrentCost, although the baud rate needs to be changed in the code - the EnviR uses a rate of 57600bps whilst the first generation panel uses 2400bps.  I believe there are also second generation panels around that look similar to the first generation ones but run at 9600bps.

Wednesday, 30 August 2017

Incompetent ISPs

Some background history

Local education authorities in the UK often operate a "broadband consortium" for the schools in the region.  The cost of providing a decent internet connection has often been very dependent on the school's geographic location - some schools can be connected up fairly cheaply whilst others are out in the sticks and very costly to connect.  So the idea is that the authority can provide internet connections to all of their schools at a fixed(ish) price to make things fairer.  Essentially, the schools that can have cheap connections subsidise those that can't.

Connecting schools into a single WAN offers a few other advantages, such as being able to centralise the internet filtering for the entire region.

As safeguarding requirements have become more stringent, the centralised filtering systems have often not kept pace and some schools have opted to buy third party filtering systems which they considered superior, instead.  So as an online safety supplier, we have picked up a number of customers who still use the authority-supplied internet connection, but buy in their own filtering system.

More recently, bandwidth requirements have been increasing at an unprecedented rate, and internet connections have been getting steadily cheaper, so the original reasons for the broadband consortia are becoming less important.

Some authorities also found that their networks couldn't cope with the increased bandwidth requirements.  Somerset was one of those authorities, with a network that was part of the South West Grid for Learning (SWGfL - operated by RM Plc), and decided to disband their network, leaving the schools to choose between contracting directly the SWGfL, or to switch to a new ISP.

For the schools that were still using the centralised filtering system, sticking with the SWGfL made sense, since they could continue more or less as-is (their internet connection itself was replaced, but the filtering system remained the same).

As an aside, I'll note that my experience of the SWGfL is that they seem to have significantly more problems than other ISPs and it takes them a lot longer to sort them out.  Part of this may be that operating a filtering system introduces a few more points of failure on their network.  But their support is done through a foreign callcentre (so you're already having to deal with a language barrier when trying to get problems sorted) and their staff don't really seem technical enough to do their jobs.  On many occasions I have been told that a problem is "fixed" when it clearly isn't and it seems they simply didn't have the technical skills to actually test it themselves.

The problem


One of our customers, who was using the Somerset connection only for internet connectivity (not filtering), asked us to help them choose which ISP to switch to.  We got them a few quotes for leased lines, but they eventually decided to stick with the SWGfL - I'm not completely clear on the costs but I'm under the impression that it wasn't terribly dissimilar to the other quotes, and the reason given for picking the SWGfL was because its "what they know".  I must admit that I don't quite understand that because as far as I can tell "what they know" is that they have had no end of problems with the SWGfL-based authority-supplied connection in the past.

Also interesting to note that most ISPs now do IPv6 as standard over leased lines (although it's still hit and miss for VDSL, etc.) but IPv6 isn't an option at all on an SWGfL connection and it doesn't sound like RM have any plans to implement it any time soon!  You also don't get a public subnet like you would with pretty much every other ISP - instead everything is done on private RFC1918 addresses and the ISP does NAT for you.  This might make sense for schools who rely on RM's filtering, but is a bit nonsensical for anyone who just wants a plain unfiltered connection.

Anyway, the school bought an unfiltered connection from the SWGfL and it was duly connected.  They paid for one of our engineers to go on site to ensure the switch over went smoothly.  One of the things we had to do was expand their external-facing IP network because RM informed the school that they needed to set up VRRP for the backup internet connection.

Very soon, the school complained about problems connecting to HTTPS websites and we did some diagnosis - a high proportion of HTTPS connections just seemed to not be successful.  This was very odd behaviour - you'd try to make a connection and the SYN packets would all just be dropped somewhere in the ISP's network, but traffic for other connections would be working just fine.  We decided the likely explanations were that the traffic was being intercepted by a broken transparent proxy, or that RM's CGNAT system was broken.

We also noticed that some proxy headers were being added to HTTP traffic, indicating that it was being transparently proxied.  This suggested that the HTTPS traffic was probably also being transparently proxied.  Transparently proxying HTTPS traffic doesn't make any sense for an unfiltered connection, and RM eventually confirmed that the traffic was indeed being directed at their filtering system.  They agreed to turn off the filtering and the school submitted a change request for this change to be made.

Several months later, the problem reoccurred and we found that the traffic was once again being directed through a transparent proxy.  The school raised a support request and RM confirmed that the traffic was being sent through their filtering system, and that the school would need to submit another change request to have this corrected.  However, RM flatly refused to turn off filtering for the whole school's network, only for a single IP address.

After an official complaint was made by the school (since RM were not supplying the "unfiltered connection" that the school had been sold), they agreed to move the school onto an unfiltered connection, but that this would require moving them onto a public IP subnet.  They allocated a /28 network and told us that the school currently had 16 public IPs allocated to 1:1 NAT rules which would need to be moved into that new /28.

We pointed out that a /28 network doesn't actually have 16 usable IP addresses (surely this was obvious and should have been taken into account when they allocated it?)  We also asked which IPs were going to be reserved for VRRP - that totally confused the issue because it turns out that, despite specifically asking us to reconfigure things so that they could use VRRP, they never configured it and don't actually need it for anything.

So eventually, RM allocated a new /27 network and we sent an engineer over for another half day to make sure things were reconfigured properly.  Of course, everything got reconfigured and the switch-over occurred and nothing worked.  After some toing and froing it turned out that they had used a completely different network than the /27 network they had allocated.  So more reconfiguration later and everything worked.

Then a few hours later, everything broke again and the school staff came in in the morning to find no internet connection.  We are assured by RM that this was a problem with their core network and nothing to do with the reconfiguration, but I certainly have my suspicions.


  • Using SWGfL may make sense for schools who want to use their filtering system, but they seem ill equipped to provide reliable unfiltered connections (I can't comment on the reliability of filtered connections, but we did see significant enough problems with their transparent proxies that I wouldn't want to vouch for the reliability of their filters).
  • RM's network seems to have significantly more problems than other ISPs.
  • They seem to take a lot longer to resolve problems than other ISPs - the school has had significant problems associated with RM's broken transparent proxies for months.
  • Dealing with foreign call centres is very frustrating, if only because you're constantly battling with a language/strong-accent barrier (I would have similar concerns with any company basing call centres in "strong accent" areas in the UK too).
  • The competence of their technical staff seems very questionable - we shouldn't be told to reconfigure stuff to allow services that they don't need and won't ever set up (VRRP), subnet allocations shouldn't spontaneously change without anyone being informed.
  • We had considered doing the second network reconfig remotely - its lucky we didn't since RM's mistake would have left the school's network unreachable from the internet.

Tuesday, 17 January 2017

Village Hotels, Subject Access Request

So following my complaint about spamming, Village Hotels responded to my Subject Access Request under the Data Protection Act.  In short, the DPA deems personal data to be owned by the person whome it regards.  A Subject Access Request allows the owner of some personal data to request information about it from a company that is holding it.

In this case I asked Village Hotels to:
  1. Confirm whether they are registered as a data controller under the Data Protection Act 1998.
  2. Provide a copy of all data they hold which relates to me.
  3. Confirm to me in writing where they obtained the data from and on what terms.
  4. Provide me with a complete list of companies, individuals, etc, that they have shared my data with and on what terms.
  5. Explain when and how they believe I gave them informed consent to send marketing emails.
They've sent some fairly extensive information, which is good, and confirmed that they are registered.

It seems that they have only my name and email address, which they thought had previously been deleted from their systems.

Notably they haven't said why they think they had consent, or where they got my data.

They enclosed an email chain, which is interesting: it includes a comment from someone at DA Group, which says
Also agree that you need to be careful emailing contacts that have had no interaction with your emails for 18 months - may be better to focus on a re-engagement campaign to these contacts rather than actively marketing to them
DA Group appear to be a marketing company, and the above comment seems a bit questionable.  The Privacy and Electronic Communications (EC Directive) Regulations 2003 don't allow unsolicited emails to be sent to anyone who hasn't given the sender informed consent.  So if they can't demonstrate that such consent has been received, they shouldn't be sending any unsolicited emails, including a "re-engagement campaign".

(Also, yes the email they sent had a confidentiality statement on the bottom, no that isn't a contract I have agreed to, so I am ignoring it.)

Friday, 13 January 2017

Village Hotels, yet again

Despite never being a customer of De Vere Venes, they sent me hundreds of spam emails, ignored my requests to stop and I eventually complained to the Information Commissioner's Office.  De Vere were instructed to stop spamming me and they admitted they had no idea where they got my details from.  Since the Privacy and Electronic Communications (EC Directive) Regulations 2003 require the sender of unsolicited emails to have obtained informed consent directly from the recipient, they're on pretty dodgy ground if they don't have any records to demonstrate that they acquired that consent.  Certainly, they won't be able to demonstrate consent with "we don't know where we got your details."

De Vere Venues then sold their brand and my details to VUR Village Hotels & Leisure.  VUR Village Hotels & Leisure used those details to start sending me spam again.  This is certainly unlawful - consent is non-transferable, so VUR Village Hotels & Leisure are in breach of the regulations by sending unsolicited emails to anyone in the database that De Vere gave them.  Furthermore, since the ICO had already told De Vere to suppress my details, actually using them was completely idiotic.  I'm also not terribly happy that De Vere sold off my data.

Much like De Vere, Village Hotels also ignored the legal notices I sent to them, so the next step was to take court action.  This is pretty easy to do - you just figure out how much the unsolicited email cost you, make sure you can justify it and submit a claim on moneyclaim.gov.  It costs £25 to file with the court.  They coughed up £225 (including the £25 filing fee) and stated that my details were bought from De Vere and that all other information had been removed during the transfer.  They didn't comment on the unlawful use of email addresses which they had bought from De Vere, beyond stating that their spam emails carry an "unsubscribe" link (which the regulations do not consider an acceptable alternative to acquiring informed consent before sending the emails).

5 days after they confirmed that they had suppressed my email address, I received another spam email.  Their defence was that it takes 3-5 days to suppress an email.  This isn't really acceptable - there's no reason that updating a database should take more than a few minutes, and the court papers were issued 10 days previous so sensibly they would have suppressed my address immediately.  In any case, as no further spams arrived I didn't press the point.

At the end of last year, it appears that VUR Village Hotels & Leisure moved to a different spam sending platform.  It looks like they hadn't actually removed my details from their database, merely flagged them as not to be used, and I guess they didn't bother to transfer that flag over to the new platform.  I started receiving more spam.  So off went another Notice Before Action.  An hour later I got an automated message from their spam sending platform (dotmailer.com) saying I'd been unsubscribed.  And 9 days later an email notifying me that they were paying another £200 compensation - I didn't even need to file with the court.

Of course, the concern is that not only do they clearly have no regard for the regulations, which prevent them from just buying a database of email addresses and spamming them, they also don't seem to have the data handling abilities required to ensure that people who unsubscribe actually remain unsubscribed.  I'd certainly recommend that any other recipients of spam from VUR Village Hotels & Leisure (or in fact, any registered UK company who is spamming your personal email address) serves them with a legal notice demanding compensation for their negligence.