Monday, 10 December 2018

Thoughts on Brexit

I find the whole Brexit thing very frustrating, and thought I would get some thoughts down to at least get things clear in my own head.  I'm going to try and keep this post free of partisan views as much as possible and just concentrate on the facts rather than debating whether or not leaving the EU is the right thing to do.

Brexit has been billed as one of the largest democratic exercises in British history.  But I think, however it turns out in the end, it will have failed to pass the "democratic" test for numerous reasons.

Due diligence requires that, before putting this to a public vote, people be well informed as to what they are voting on and the expected impact of each of the options.  Whilst forecasts are often not completely accurate, it does seem reasonable to at least make an attempt at forecasting the impact of each option.  Unfortunately no forecasts were produced by the government before the referendum, so the idea that people (on either side) were making an informed choice seems a bit bonkers to me.

Over the past 2 years, I've seen lots of claims that people "obviously" knew exactly what they were voting for, but if we take a critical look at what was said prior to the referendum, this seems like a very revisionist view to me.
  • In 2013, Nigel Farage, a prominant figure in the Leave.EU campaign, was advocating that the UK's relationship with the EU should be like Switzerland's.  Like the UK, Switzerland are part of the European Economic Area, and unlike the UK they are part of the Schengen Area and therefore do not operate passport checks on their boarders.  Due to their participation in the Schengen Area, in many ways they have less control over EU migration than the UK does.
  • The official "Vote Leave" campaign, along with various politicians who were involved in that campaign such as Daniel Hannan and Boris Johnson explicitly said that, following a vote to leave the EU, the UK would still have access to the EEA ("single market").
The referendum question was "Should the United Kingdom remain a member of the European Union or leave the European Union" and these days we're being told that this was a simple question and that everyone understood that leaving the EU also meant leaving the EEA.  It sounds pretty suspect to me to claim that everyone understood this to mean the exact opposite of what most of the campaigners were saying it meant.

Anyway, so referendum done and the result was 52:48%, which is a pretty marginal thing to do.  The country was very divided, so the government needed to lead the way and bring the country back together again.  So in order to do that they told 48% of the population that their views would be completely ignored.  Disenfranchising almost half the population by writing off their views entirely doesn't sound like the best way of running the country, and seems to me to have set the government on a course that they can't possibly win.

Still having done no impact assessments and having no idea how to negotiate a withdrawal agreement, parliament then triggered Article 50 and started the UK's withdrawal.  The government's view seemed to be that we could just demand whatever we wanted and the EU would be forced to comply.  Now that it has become clear that this isn't going to work, the EU are often being painted as bullies, which is something I don't understand - it is the EU's job to look after the best interests of the remaining members, not to cut us a deal that is not in their interest.  The fact that the EU is not cutting the UK a great deal simply demonstrates the weakness of our negotiating position and has nothing to do with "bullying" - if the UK had a negotiating position that was as strong as the government lead us to believe, "bullying" from the EU would not be possible.

In the lead up to the referendum, both sides of the debate put forward a number of arguments that, at best could be considered very misleading, and at worst outright lies.  For example, the £350M/week claim that was made by the official Vote Leave campaign - Dominic Cummings, director of the campaign said all of their research indicated that they couldn't have won if they hadn't made that claim.  On top of that, campaigns on both sides of the debate have been found guilty of breaking electoral spending rules, and independent research conducted by Professor Philip Howard suggests that the illegal spending could well have tipped the balance in favour of leave winning.

I don't expect the government to cancel Brexit outright on the first sniff of misconduct during the referendum, but I do expect them to conduct research to determine whether the misconduct could have influenced the result.  So far, the government has roundly refused to acknowledge any possibility that the referendum result could have been influenced by any of this behaviour, whilst failing to actually present any evidence to support their position.

Now, assuming we're going to leave, lets examine how we can do so.

The UK joined the EEC in 1975 and over the following 40 years, numerous other agreements have been build on top of this foundation in small increments.  To my mind, the idea that you can completely remove this foundation overnight without everything that was build on top of it collapsing seems like nonsense.  The only sane way I can see to leave the EU without catastrophic collateral damage, is to leave in the same way as we entered: Start by leaving the EU but remaining part of the EEA and the various agencies and agreements that we participate in.  Then spend the next 30-40 years incrementally unpicking the tangle of agreements to gradually extricate ourselves from the parts of the EU that we want to be rid of.

There are a few, very vocal, people shouting "just leave".  Promoting the idea that we should just bomb out of the EU without any kind of an agreement to ensure an orderly withdrawal.  They remind me of some customers I have to deal with who, when faced with a problem just yell "I don't care, just fix it" despite us making protestations that "it's quite complicated".  When dealing with those customers, I can always guarantee that any problems created by "just fixing it" will be considered our fault rather than the fault of the customer for refusing to listen when we tried to explain what collateral damage may be caused.  The people shouting "just leave" never seem to be able to present a coherent idea about how they can mitigate the myriad of agreements that will inevitably collapse when their foundations are pulled from under them.  The most obvious is of course the Good Friday Agreement, the collapse of which could well lead to a reformation of the IRA, regular bombings and the breakup of the UK.

So there we go, however this turns out, this will be an utter failure of democracy:
  • Information regarding the impact of leaving was not made available to the public prior to the referendum, so there was no chance of an informed choice being made by the public.
  • Information regarding the impact of leaving wasn't made available to parliament, so there was no chance of an informed choice being made by MPs.
  • There is no way to know from the referendum result what the people who voted leave actually wanted in terms of things like EEA membership.  Never the less, the result is being used to support a course of action which is the exact opposite of what the campaigners said that voting "leave" would mean.
  • No matter how much evidence is presented to support it, any suggestion of the result being influenced by any wrongdoing is being shouted down whilst no counter-evidence is given.

Tuesday, 18 September 2018

Chasing wild geese

So, a quick one this morning.  One of our customers has been having problems accessing Lloyds Bank's corporate payments gateway.

The first thing they did was phone Lloyds (very sensible).  Lloyds told them that there were no problems at their end and to clear cookies, add the site to the ActiveX trusted sites list (seriously, why is anyone using ActiveX these days?), etc.  Still not working, so must be a problem with the customer's firewall.

So the customer phoned us.  We pointed a browser at https://payments.corporate.lloydsbank.com/ (on an independent internet connection) and nothing happens - just sits there waiting.  So clearly Lloyds are having problems.

Lets try some slightly lower level debugging:
# openssl s_client -connect payments.corporate.lloydsbank.com:443 -servername payments.corporate.lloydsbank.com
And it just sits there...  Eventually:
CONNECTED(00000003)
And more sitting there doing nothing...  Then eventually:
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
Oh dear...  I should've got a certificate back from that but instead the web server dropped the connection.  So a few of obvious problems to look into here:
  1. It took a really long time for "CONNECTED" to appear...
  2. It took a really long time for anything to happen once we're connected...
  3. Finally, the web server failed to send us a certificate and negotiate an encrypted TLS session.
Firing up tcpdump, I found that:
  1. There was a significant delay before the first packet (SYN) even appeared.  So something else was going on before the connection was even attempted.  A DNS problem was a good bet.
  2. The first packet (SYN) was resent about 3 times before the web server responded.  This would also cause a significant delay in starting the connection.
So, investigating a potential DNS problem:
dig payments.corporate.lloydsbank.com
Resulted in a long wait before failing - definitely a DNS problem then.  Lets find the name servers responsible:
# dig payments.corporate.lloydsbank.com ns +trace
<snip>

payments.corporate.lloydsbank.com. IN NS  ns-lv6.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv2.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv7.lloydsbanking.com.
payments.corporate.lloydsbank.com. IN NS  ns-lv3.lloydsbanking.com.
Looking those up gives me:
ns-lv6.lloydsbanking.com. IN    A    141.92.88.1
ns-lv2.lloydsbanking.com. IN    A    141.92.96.1
ns-lv7.lloydsbanking.com. IN    A    195.171.195.169
ns-lv7.lloydsbanking.com. IN    A    195.171.195.168
ns-lv7.lloydsbanking.com. IN    A    195.171.195.167
ns-lv7.lloydsbanking.com. IN    A    195.171.195.166
ns-lv7.lloydsbanking.com. IN    A    195.171.195.165
ns-lv7.lloydsbanking.com. IN    A    195.171.195.164
ns-lv7.lloydsbanking.com. IN    A    195.171.195.163
ns-lv7.lloydsbanking.com. IN    A    195.171.195.170
ns-lv3.lloydsbanking.com. IN    A    141.92.104.1
So there are 11 name servers.  And by trying to look up against each of those name servers I find that 9 of them are down.  That means that about 82% of DNS requests are going to time out - at best things are going to be very slow while the customer's DNS server makes repeated DNS lookups and waits for each to time out; at worst it will fail to find a working DNS server and give up, rendering the website inaccessible.

To summarise:
  • 9 out of 11 of Lloyds' DNS servers were down, resulting in intermittently very slow or even completely broken DNS lookups.
  • If you managed to resolve the web server's IP address, it took a long time to accept the connection.
  • If you managed to get a connection, the web server may fail to negotiate an encrypted TLS session with the client.
With multiple Lloyds Bank servers having serious problems, I wouldn't mind betting that they are being attacked.  But why didn't Lloyds' own support people know / admit that there were problems on their end rather than sending the customer on a wild goose chase - it only took us 15 minutes to diagnose a problem that Lloyds' own people must have already known about.

Tuesday, 31 July 2018

Why do they not listen?

I don't usually talk much about our customers, but sometimes things happen which truly beggar belief.

For many years we have been contracted by a consortium of schools who were geographically close and originally wanted to be able to share a single connection for financial reasons.  This is quite a common arrangement.  Due to the shifting landscape of internet provision, costs and politics, the arrangement came to an end some time ago, which is fine - projects don't last forever, customers come and go and the project ended amicably, with all of the schools involved being pretty happy with us.  In fact, they all ended up taking out independent contracts with us for one thing or another after the project ended anyway.

Just one of the schools has been somewhat... "odd" at times though (I will refer to them simply as "School" to retain their anonymity), and despite our best efforts it has gradually caused problems for them.  Their ICT support is outsourced, which can be both good or bad - some of the companies that provide outsourced ICT services are pretty good, but some of them seem to have a "rip everything out" attitude and insist on unnecessarily replacing a school's equipment on day one instead of spending some time looking to see what is working well and what isn't and only replacing the stuff that isn't working.  This generally seems to be because they want to use systems that they are already used to rather than learning something new.  There's some merit in trying to standardise on systems you know, but obviously leads to a lot of disruption and expense for the school, so in my view is not a great way of doing things.

Anyway, the story probably starts in 2013.

Summer 2013

We had been providing the connectivity between the schools for some years by this point.  Because of limitations of the technology which was available at the time when it was installed, the schools had unreliable, but redundant, interconnects.  These weren't installed by us, but we were contracted to provide and maintain systems to use those unreliable interconnects to provide reliable connectivity.  We were also contracted to provide online safety systems (web filtering, etc.) to the whole consortium.

School: We've just changed our ICT provider and the new provider has decided to replace the existing online safety system with a third party system.
Us: That's fine, but the connections you use for internet access aren't reliable enough to use independently and the equipment you're proposing to remove is used to provide reliable internet access over those unreliable connections. The third party equipment that you want to install is incompatible with the protocols used by the existing equipment, which means you will also need to replace the equipment at the far end of the connections.

Obviously we prefer not to lose a customer, but if they want to switch to another provider then that's fine and we try to guide them and minimise the disruption as best we can.

The ICT provider ordered the third party system, plugged it in, discovered that it didn't work with the interconnects (as we had told them) and ended up backtracking on the whole thing.  So they left our online safety system in place and in fact I think their ICT provider probably decided it was ok in the end - at least they made no further moves to replace it.

Summer 2015

School changed ICT provider again.  This didn't really have any impact on us.

Summer 2016

By now, all of the schools in the consortium, except for School had installed their own internet connections and the interconnects were just being used for a small amount of local traffic and as redundancy in case of failure of one of the internet connections.  The consortium decided that the project had run its course and announced that it would be dismantled by summer 2017.

There was also some discussion about retiring the infrastructure early:

Consortium: We should simplify things by removing all of the equipment that is managing the connections immediately.
Us: We can't do that because that equipment is still need to provide the connection to School.
Consortium: No problem, we'll leave it as-is until summer 2017.

Spring 2017

School: We intend to continue using the existing interconnects after the summer.
Us: It isn't economic to do so since you're now footing the whole bill instead of it being shared by the whole consortium.  The existing equipment is also very old so liable to fail soon.  Replace everything with a new connection, it'll cost less than continuing with the existing equipment.  We can do this for you or you can get a third party in, we don't mind either way.

Summer 2017:

Us: As agreed last year, the existing interconnects will now be shut down.  School will need to migrate over to their new connection.

School: No alternative connection has been procured, there's no time left to get one now, we need to keep using the existing connections.
Us: We already said this was uneconomic, but as a good will gesture we'll take some of the hit ourselves and knock 50% off the cost.  But this is a one year only deal - we will not support this next year because the equipment is well past its end of life.  Also, as the equipment is very old, we recommend you follow our original recommendation and replace the interconnect ASAP since it might fail at any point.  If any of the hardware fails, we won't fix it.

Spring 2018:

Us: Just a reminder, you need to replace the interconnects ASAP.

Summer 2018:

School: We've just changed our ICT provider (again) and the new provider has decided to replace the existing online safety system with a third party system.
Us: That's fine.  As you already know, the connection you are using for your internet access is going out of service this summer, we presume you've procured a replacement?
School: No we haven't, we intend to continue using the existing interconnects.
Us: But those interconnects aren't reliable enough to use independently and the equipment you're proposing to remove is used to provide reliable internet access over those unreliable connections. The third party equipment that you want to install is incompatible with the protocols used by the existing equipment, which means you will also need to replace the equipment at the far end of the connections.
Us: In fact, this is exactly what we said in the summer of 2013, then again in the summer of 2016, then again in spring 2017, and in summer 2017, and in spring 2018.
Us: Also, we already told you a year ago that we weren't going to support any of this equipment which manages those interconnects any more as it is far too old.
School: We don't need that equipment, we're just going to use a single (15 year old, unreliable) connection in isolation.
Us: Errm, that will be really unreliable, here are some statistics from our monitoring data to show just how unreliable it will be.

Panic ensues at School.

School: Why didn't you tell us that we would need a new connection!  It's now far too late to procure one in time.  You must extend our contract for free.
Us: Umm, no.
Us: We've made every effort to recommend the most economic and reliable way forward and have been ignored at every step.
Us: Last year we dug you out of a hole you'd made for yourselves and even gave you a big cost reduction out of our good will.  You have repaid us by continuing to ignore our recommendations, blaming us for the mess you've got yourself in, paying your last invoice months late and cancelling your contract with us.
Us: You have now demanded that we dig you out of a hole again out of the goodness of our hearts at extremely short notice.
Us: Here are a selection of get-out-of-jail cards from our standard price list, which we are happy for you to buy from us at the standard price, go pick one.

And apparently this is all our fault...  At least now that the contract with them has ended we won't have to deal with any of the fallout from this mess.  Seems like a classic case of "I think we've heard enough from the experts" to me :)

Thursday, 5 April 2018

Thoughts on the gender pay gap

I've already managed to upset someone on Facebook because I apparently said that women are worth less than men (I didn't).  I don't regard myself as a feminist, but rather an equalist.  Discrimination of all forms is bad - women shouldn't be discriminated against based on their gender, but similarly men shouldn't be discriminated against in order to give women an advantage.

Discrimination is as old as time itself but I don't think you fix the problem by just changing which group you discriminate against, any more than you prevent war by changing which group of innocent people you're blowing up.

The British government is now requiring all businesses with over 250 employees to publish figures summarising the gender pay gap.  There are a few bits of information required, but essentially this boils down to a simple average difference in pay between men and women.

The press is then using the figures to bash the companies with the biggest pay gap.  No consideration is being given to what types of work are being done since that isn't in the information that companies are expected to publish.  i.e. this is not a like-for-like comparison.

There are a few potential reasons why women may be earning less than men:
  1. Maybe there are less women qualified and applying for the higher paid jobs.
  2. Maybe employers are refusing to employ women in the higher paid jobs.
  3. Maybe employers are employing women in the higher paid jobs but are paying them less than the men who are doing the same work.
All of these points are, of course, a problem.  The two last points are directly under the employer's control and any employer undertaking in this kind of discrimination deserves to have the book thrown at them - I firmly believe that equally qualified men and women should be given the same opportunities as each other and the same pay for doing the same work.

However, it is fairly unclear to me how the first point can be regarded as a specific employers' fault.  Since the information that is being made available doesn't do anything to differentiate between these possibilities, it seems completely unfair to vilify an employer based solely on this data.

So we have STEM companies and the construction industry with a fairly big pay gap simply because it's very difficult to recruit women to do the higher paid jobs in these fields.  I'm sure that in some cases there is discrimination going on, but you can't determine that from the data being used by the press to attack employers.

To demonstrate the issue, lets take a simple fictional employer - they aren't discriminating and they have the following breakdown of employees:

Total Men Women Salary
Engineers 200 180 20 £50,000.00
Admin staff 50 15 35 £20,000.00
Total 250 195 55

The mean pay is £47,692.31 for men and £30,909.09 for women, yielding a pay gap of £16,783.22, even though everyone doing the same job is paid the same.

So, how can the employer fix their pay gap?  Since we're already assuming there is no discrimination going on, we can look at it rationally with maths:

1. Increase the proportion of female engineers
It's very unclear to me how a single employer can increase the proportion of applicants who are female.  In order to do this the proportion of women training in engineering needs to be increased, starting with school kids.  There is some scope for the industry as a whole working to promote engineering to women, but it takes years and a single employer can't do a lot on their own.

At work, when we were last recruiting, we didn't end up hiring a man because we're horrible people who support the patriarchy; we ended up hiring a man because no women applied for the job.  We would usually want to pick the best person for the job, regardless of their gender.

So with far less women in the engineering job market than men, the immediate options are:
  • Increase the women's benefits and decrease the men's.  The women will be "overpaid" with respect to other employers and want to work for you whilst the men will be "underpaid" and get a job elsewhere.
  • Recruit underqualified women to make up the numbers, since there aren't enough qualified women applying.  Recruiting people who aren't qualified to do the job sounds like a bad idea for the business.
  • Restrict the number of applications from men.
All of these options discriminate against men based solely on their gender and are therefore pretty unethical (and probably illegal).

2. Increase the proportion of male administrative staff
So given that we probably can't do a lot to recruit more female engineers, we could tweak the balance elsewhere in the business.  The average women's pay is being dragged down by the fact that a disproportionate number of women are employed in the lower paid administrative roles.  Only 10% of the engineers are women, but 70% of the administrative staff are women.  If the employer reduces the proportion of women in the admin roles down to 10%, that will eliminate the pay gap.

This problem is pretty much the opposite of (1) - the same options apply, but this time the employer must discriminate against women.  Again, doesn't strike me as a good plan.

3. Increase the women's pay or decrease the men's pay
So far, the employer has paid men and women the same amount for the same work, which has led to a big pay gap simply because of the job role demographics.

Adding about 55% to the womens' pay in both job roles eliminates the pay gap.  We're now paying female engineers £77,000 and female administrative staff £30,800.  Of course, the employer may well not be able to afford these kinds of expenses, especially when the men find out that they are earning far less than the women and take their employer to court for sexual discrimination.

Similarly, reducing the men's pay by 65% across the board achieves a similar result - the employer gets sued into the ground for sexual discrimination, and if they survive that, with the men now being paid far below the market rate, they all leave for greener pastures.

4. Pay the engineers and administrative staff the same
Another option is to decrease the pay the engineers receive and increase the pay the administrative staff receive.  By paying everyone £44,000 irrespective of what job they do, the pay gap is eliminated whilst keeping the total wage bill the same.

Unfortunately, the company's engineers are now underpaid relative to the market rate, so they will leave and it will be impossible to recruit replacements.

The entire industry could follow suit, but this would lead to a long term shortage in engineers - it costs tens of thousands of pounds and several years to become an engineer, and how many people would do that if their pay is the same as someone who hadn't spent that time and money on training?


I am a firm believer that equally qualified and experienced women and men are equally valuable and should be paid the same.  For some jobs it is easier than others to ensure that this is happening - where there are fixed non-negotiable pay scales things are obviously clearer than jobs where employees are expected to negotiate their salaries.  In the latter case, people who are poor negotiators are obviously not going to do as well as good negotiators.  I don't know if there's a gender bias when it comes to negotiating skill, but any protection should surely extend far beyond the work place since poor negotiators also lose out when negotiating other things, such as buying a car, etc.

There are also big differences in demographics that need to be accounted for: is a 35 year old who graduated from university at 21 and has been in relevant employment for the whole time (giving them 14 years industry experience) as valuable as someone else who took 10 years off between 23 and 33 to bring up their children (so they have only 4 years industry experience)?  These people could be either men or women, but in the current society the majority of child carers are women and it doesn't seem right to completely discount that when comparing pay.

From taking a rational look at the data being collected, it seems clear to me that it is tackling the wrong thing.  Whilst I'm sure that discrimination is happening, the data being published cannot be used to determine who is discriminating.  Indeed, in many cases reducing the pay gap seems to actually require the employer to discriminate, so the whole thing seems very counter productive to me.

The things that need to be tackled are:
  • Ensuring that men and women are given the same opportunities.
  • Ensuring that men and women receive similar pay for similar work.
  • Ensuring that neither men and women are put off from taking any opportunities that are open to them.
Lets work on those points instead of attacking employers for things that they can't do much about.

Monday, 30 October 2017

ESP8266 CurrentCost Wifi Gateway

I have two CurrentCost power monitors - a CC128 EnviR panel in the living room which monitors our entire electricity usage, and an old white first generation panel in the office which monitors the office's electricity usage.

The CurrentCost panels output real time data in XML format every 6 seconds through a TTL serial connection presented as an 8P8C connector on the bottom, and I used to have a WRT54GL wireless access point connected to that, which sent the data to the server.  The WRT54GL was retired some time ago, so I wanted another way of recording the power usage data.  The ESP8266 seemed like a good choice to provide a bridge to my wifi network and I've chosen to use an ESP-12F board.  I could build a small enough device on stripboard to sit in the cavity under the base of the panel and connect to the CurrentCost's TTL serial output.


The 8P8C connector also includes regulated and unregulated power connections, but the 3v PSU is only rated for 80mA which is nowhere near enough to power the ESP8266.  I didn't want to have a second PSU, so opted to use a single USB power supply to power everything.  I've used an LM1117T LDO regulator to drop the supply voltage down to 3.3v, and connected this to both the ESP8266 and the unregulated supply rail of the CurrentCost.  There's a schottky diode between the regulator and the CurrentCost - this isn't strictly necessary, but will protect against someone plugging the CurrentCost PSU in at the same time as the wifi bridge.

GPIO0 and GPIO2 are pulled up by 10KΩ resistors, GPIO15 is pulled down by a 10KΩ resistor - you can probably get away with omitting the resistors and connecting these directly to Vcc / 0v respectively, although using a resistor to pull up GPIO0 allows the ESP8266 to be programmed in circuit by temporarily pulling it down to 0v.

CH_PD and reset are pulled up all the time by tieing them to the Vcc rail.

The connections to the CurrentCost's 8P8C connector are:
  • Pin 1 - Vcc / unregulated power
  • Pin 4 - Ground / 0v
  • Pin 8 - TTL serial data
Component-side view on the left, flipped for a track-side view on the right
Since the ESP-12F has solder pads with a 2mm spacing and the stripboard has 0.1" hole spacing, the ESP has to stand off the stripboard a little.  I've taken the opportunity to use the space between the ESP-12F and the board for the pullup / pulldown resistors.  I've shown the ESP-12F as a translucent block on the stripboard layout above so you can see the components under it.

The power connector is a surface mount mini-USB socket - obviously that's incompatible with stripboard, so I have just tacked on a couple of wires to connect it to the circuit and soldered its mounting lugs down to the track-side.  I find it useful to hot-glue the wires onto the board for mechanical support to avoid breaking off the socket's pins.
Component-side view

Track-side view
Although the CurrentCost outputs XML, I'm not bothering to parse that in the ESP8266.  Helpfully, each XML sentence is terminated with a newline, so it's easy to use Serial.readStringUntil('\n') to grab the whole sentence and post it to the web server.  Data posted over HTTP needs to be URL encoded and there doesn't seem to be a standard library to do this, so I ended up writing a quick and dirty encoder.  A PHP script on the web server parses the XML and feeds the data into RRDTool, and I'm using Cacti to display graphs of the data.




I wanted my graphs to display a total number of kilowatt hours used.  Unfortunately Cacti doesn't have a generic function to do this.  It can do bandwidth summation for network usage (i.e. displaying a "total megabytes" on a "bytes per second" graph), which is almost what I need.  Using that on the "watts" data gives a total number of watt seconds (i.e. joules) used, but no way to divide this by 3600 to produce watt hours.  In the end I added a COMPUTE data source to the RRD file, which records watts divided by 3600, and Cacti can then use that to calculate the total watt hours of electricity consumed.  Since I'm abusing the bandwidth summation functions, this unfortunately adds a "B" (bytes) unit to the total displayed, but I can live with that.

The code, schematic and layout are available from my Subversion repository.  The circuit works for both the CurrentCost EnviR (also known as the CC128) and the first generation CurrentCost, although the baud rate needs to be changed in the code - the EnviR uses a rate of 57600bps whilst the first generation panel uses 2400bps.  I believe there are also second generation panels around that look similar to the first generation ones but run at 9600bps.


Wednesday, 30 August 2017

Incompetent ISPs

Some background history


Local education authorities in the UK often operate a "broadband consortium" for the schools in the region.  The cost of providing a decent internet connection has often been very dependent on the school's geographic location - some schools can be connected up fairly cheaply whilst others are out in the sticks and very costly to connect.  So the idea is that the authority can provide internet connections to all of their schools at a fixed(ish) price to make things fairer.  Essentially, the schools that can have cheap connections subsidise those that can't.

Connecting schools into a single WAN offers a few other advantages, such as being able to centralise the internet filtering for the entire region.

As safeguarding requirements have become more stringent, the centralised filtering systems have often not kept pace and some schools have opted to buy third party filtering systems which they considered superior, instead.  So as an online safety supplier, we have picked up a number of customers who still use the authority-supplied internet connection, but buy in their own filtering system.

More recently, bandwidth requirements have been increasing at an unprecedented rate, and internet connections have been getting steadily cheaper, so the original reasons for the broadband consortia are becoming less important.

Some authorities also found that their networks couldn't cope with the increased bandwidth requirements.  Somerset was one of those authorities, with a network that was part of the South West Grid for Learning (SWGfL - operated by RM Plc), and decided to disband their network, leaving the schools to choose between contracting directly the SWGfL, or to switch to a new ISP.

For the schools that were still using the centralised filtering system, sticking with the SWGfL made sense, since they could continue more or less as-is (their internet connection itself was replaced, but the filtering system remained the same).

As an aside, I'll note that my experience of the SWGfL is that they seem to have significantly more problems than other ISPs and it takes them a lot longer to sort them out.  Part of this may be that operating a filtering system introduces a few more points of failure on their network.  But their support is done through a foreign callcentre (so you're already having to deal with a language barrier when trying to get problems sorted) and their staff don't really seem technical enough to do their jobs.  On many occasions I have been told that a problem is "fixed" when it clearly isn't and it seems they simply didn't have the technical skills to actually test it themselves.

The problem

 

One of our customers, who was using the Somerset connection only for internet connectivity (not filtering), asked us to help them choose which ISP to switch to.  We got them a few quotes for leased lines, but they eventually decided to stick with the SWGfL - I'm not completely clear on the costs but I'm under the impression that it wasn't terribly dissimilar to the other quotes, and the reason given for picking the SWGfL was because its "what they know".  I must admit that I don't quite understand that because as far as I can tell "what they know" is that they have had no end of problems with the SWGfL-based authority-supplied connection in the past.

Also interesting to note that most ISPs now do IPv6 as standard over leased lines (although it's still hit and miss for VDSL, etc.) but IPv6 isn't an option at all on an SWGfL connection and it doesn't sound like RM have any plans to implement it any time soon!  You also don't get a public subnet like you would with pretty much every other ISP - instead everything is done on private RFC1918 addresses and the ISP does NAT for you.  This might make sense for schools who rely on RM's filtering, but is a bit nonsensical for anyone who just wants a plain unfiltered connection.

Anyway, the school bought an unfiltered connection from the SWGfL and it was duly connected.  They paid for one of our engineers to go on site to ensure the switch over went smoothly.  One of the things we had to do was expand their external-facing IP network because RM informed the school that they needed to set up VRRP for the backup internet connection.

Very soon, the school complained about problems connecting to HTTPS websites and we did some diagnosis - a high proportion of HTTPS connections just seemed to not be successful.  This was very odd behaviour - you'd try to make a connection and the SYN packets would all just be dropped somewhere in the ISP's network, but traffic for other connections would be working just fine.  We decided the likely explanations were that the traffic was being intercepted by a broken transparent proxy, or that RM's CGNAT system was broken.

We also noticed that some proxy headers were being added to HTTP traffic, indicating that it was being transparently proxied.  This suggested that the HTTPS traffic was probably also being transparently proxied.  Transparently proxying HTTPS traffic doesn't make any sense for an unfiltered connection, and RM eventually confirmed that the traffic was indeed being directed at their filtering system.  They agreed to turn off the filtering and the school submitted a change request for this change to be made.

Several months later, the problem reoccurred and we found that the traffic was once again being directed through a transparent proxy.  The school raised a support request and RM confirmed that the traffic was being sent through their filtering system, and that the school would need to submit another change request to have this corrected.  However, RM flatly refused to turn off filtering for the whole school's network, only for a single IP address.

After an official complaint was made by the school (since RM were not supplying the "unfiltered connection" that the school had been sold), they agreed to move the school onto an unfiltered connection, but that this would require moving them onto a public IP subnet.  They allocated a /28 network and told us that the school currently had 16 public IPs allocated to 1:1 NAT rules which would need to be moved into that new /28.

We pointed out that a /28 network doesn't actually have 16 usable IP addresses (surely this was obvious and should have been taken into account when they allocated it?)  We also asked which IPs were going to be reserved for VRRP - that totally confused the issue because it turns out that, despite specifically asking us to reconfigure things so that they could use VRRP, they never configured it and don't actually need it for anything.

So eventually, RM allocated a new /27 network and we sent an engineer over for another half day to make sure things were reconfigured properly.  Of course, everything got reconfigured and the switch-over occurred and nothing worked.  After some toing and froing it turned out that they had used a completely different network than the /27 network they had allocated.  So more reconfiguration later and everything worked.

Then a few hours later, everything broke again and the school staff came in in the morning to find no internet connection.  We are assured by RM that this was a problem with their core network and nothing to do with the reconfiguration, but I certainly have my suspicions.

Conclusions


  • Using SWGfL may make sense for schools who want to use their filtering system, but they seem ill equipped to provide reliable unfiltered connections (I can't comment on the reliability of filtered connections, but we did see significant enough problems with their transparent proxies that I wouldn't want to vouch for the reliability of their filters).
  • RM's network seems to have significantly more problems than other ISPs.
  • They seem to take a lot longer to resolve problems than other ISPs - the school has had significant problems associated with RM's broken transparent proxies for months.
  • Dealing with foreign call centres is very frustrating, if only because you're constantly battling with a language/strong-accent barrier (I would have similar concerns with any company basing call centres in "strong accent" areas in the UK too).
  • The competence of their technical staff seems very questionable - we shouldn't be told to reconfigure stuff to allow services that they don't need and won't ever set up (VRRP), subnet allocations shouldn't spontaneously change without anyone being informed.
  • We had considered doing the second network reconfig remotely - its lucky we didn't since RM's mistake would have left the school's network unreachable from the internet.

Tuesday, 17 January 2017

Village Hotels, Subject Access Request

So following my complaint about spamming, Village Hotels responded to my Subject Access Request under the Data Protection Act.  In short, the DPA deems personal data to be owned by the person whome it regards.  A Subject Access Request allows the owner of some personal data to request information about it from a company that is holding it.

In this case I asked Village Hotels to:
  1. Confirm whether they are registered as a data controller under the Data Protection Act 1998.
  2. Provide a copy of all data they hold which relates to me.
  3. Confirm to me in writing where they obtained the data from and on what terms.
  4. Provide me with a complete list of companies, individuals, etc, that they have shared my data with and on what terms.
  5. Explain when and how they believe I gave them informed consent to send marketing emails.
They've sent some fairly extensive information, which is good, and confirmed that they are registered.

It seems that they have only my name and email address, which they thought had previously been deleted from their systems.

Notably they haven't said why they think they had consent, or where they got my data.

They enclosed an email chain, which is interesting: it includes a comment from someone at DA Group, which says
Also agree that you need to be careful emailing contacts that have had no interaction with your emails for 18 months - may be better to focus on a re-engagement campaign to these contacts rather than actively marketing to them
DA Group appear to be a marketing company, and the above comment seems a bit questionable.  The Privacy and Electronic Communications (EC Directive) Regulations 2003 don't allow unsolicited emails to be sent to anyone who hasn't given the sender informed consent.  So if they can't demonstrate that such consent has been received, they shouldn't be sending any unsolicited emails, including a "re-engagement campaign".

(Also, yes the email they sent had a confidentiality statement on the bottom, no that isn't a contract I have agreed to, so I am ignoring it.)