Finding memory over-usage

Memory leaks are a common problem when writing software - this is where the software has asked the operating system (OS) for a lump of memory, but has forgotten to tell the OS when it finishes using that memory.

Tracking down memory leaks can be a real pain, but software such as Valgrind Memcheck helps - Valgrind lets you run the software (very very slowly!) and keeps track of when memory is allocated and freed.  When the software exits, Valgrind gives you a summary of memory that still hasn't been freed.

However, there is another problem very similar to a memory leak - if the memory has been allocated, but is kept in use by your software for far longer than necessary.  Eventually the software will release the memory again, so technically this isn't a "leak", but the effect is very similar.  Since the software is holding onto memory allocations for a long time, it may require a lot more of the computer's memory than you would expect.  On shut down, the software would usually release all of these memory allocations, so the usual method of tracking down leaks (i.e. see what hasn't been released at the end) isn't going to work.

But we can still use Valgrind, and rather than wait until the program exits we can get a list of currently allocated memory at any point while the program is running.  Obviously this includes everything that has been allocated, not just the "problem" allocations.  However, if the problem allocations amount to a very significant amount of memory, this tends to stick out like a sore thumb.

First of all run the program to be debugged inside Valgrind with the --vgdb=full commandline argument:

valgrind --vgdb=full --track-fds=yes --tool=memcheck --leak-check=full --show-reachable=yes --leak-resolution=high --num-callers=40 ./some_executable

In another window, we can then fire up the the gdb debugger:

gdb some_executable

At the gdb prompt, the following will connect to the running process and obtain the list of allocations:

target remote | vgdb

set logging file /tmp/allocations.txt

set logging on

monitor leak_check full reachable any

The memory allocations will be logged to /tmp/allocations.txt.  You can do this several times while the program is running and then compare the outputs. 

[Edit:  You can use vgdb directly, instead of gdb: "vgdb leak_check full reachable any"]

Debugging IPSEC

Today I had occasion to do some IPSEC debugging.  There were multiple tunnels between two endpoints (using OpenS/Wan's "leftsubnets={...}" and "rightsubnets={...}" declarations) and one of the tunnels was reported as being dead.  The remaining tunnels were carrying a lot of traffic and couldn't be shut down.

Usually I'd ping some stuff on the other side of the VPN and tcpdump the connection to see if the pings are being encapsulated.  But with multiple tunnels between the same endpoints, just filtering the traffic by source and destination address doesn't really help - you'd end up capturing traffic for all of the tunnels.

Each tunnel has its own pair of identifiers (SPIs) - one for each direction.  So first thing to do is run "ip xfrm policy show" and you get something like:

# ip xfrm policy show
src 10.0.0.0/24 dst 203.0.113.1
    dir out priority 2343 ptype main
    tmpl src 198.51.100.1 dst 203.0.113.1
        proto esp reqid 16389 mode tunnel
src 192.168.0.0/24 dst 10.0.0.0/24
    dir fwd priority 2343 ptype main
    tmpl src 203.0.113.1 dst 198.51.100.1
        proto esp reqid 16389 mode tunnel
src 192.168.0.0/24 dst 10.0.0.0/24
    dir in priority 2343 ptype main
    tmpl src 203.0.113.1 dst 198.51.100.1
        proto esp reqid 16389 mode tunnel

(The output will actually be much longer if you've got multiple tunnels).  From this we can extract the reqid from the tunnels we care about - 16389 in this case.

Now we do "ip xfrm state show":

# ip xfrm policy show
src 203.0.113.1 dst 198.51.100.1
    proto esp spi 0x01234567 reqid 16389 mode tunnel
    replay-window 32 flag 20
    auth hmac(sha1) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    enc cbc(des3_ede) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
src 198.51.100.1 dst 203.0.113.1
    proto esp spi 0x89abcdef reqid 16389 mode tunnel
    replay-window 32 flag 20
    auth hmac(sha1) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    enc cbc(des3_ede) 0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

(Again, the output will be much longer than this).  Now look up the reqid and make a note of the associated spis (0x01234567 and 0x89abcdef).


Finally, we can ask tcpdump to only show us traffic that matches those SPIs:

# tcpdump -n -i internet 'ip[20:4] = 0x01234567' or 'ip[20:4] = 0x89abcdef'

In this case I found that the encapsulated traffic was being transmitted ok, but we weren't receiving any from the other end.  Turns out the other side's firewall was dropping their traffic.

Ranting about LEA Network Administrators

I'm getting increasingly tired of the network administrators at a certain LEA.  I'm going to venture that they aren't really qualified to run the LEA's WAN...

Back at the start of July, one of our customers reported that an application was intermittently extremely slow or completely failed.  Originally the customer thought that it was a firewalling problem, but we identified a DNS problem as the cause - the LEA's DNS server was taking a few orders of magnitude longer than you'd expect to respond to AAAA record lookups for two domains that were used by the app, and eventually responded with a failure.

A quick explanation is probably in order here: when a client needs to connect to a web server on the internet, it has to convert the domain name (e.g. www.example.com) into the numerical IP address(es) of the server(s).  It does this through the Domain Name System (DNS).  Typically a (modern) client requests both "A" records and "AAAA" records from a DNS server - "A" records list the (legacy) IPv4 addresses for the web server, whilst "AAAA" records list its (newer) IPv6 addresses.  A web server may not have both IPv4 and IPv6 addresses, but the DNS server still has to produce a successful response to tell the client this.

Importantly, even if the client only has a legacy IPv4 internet connection, it may not know that it can't contact a server using IPv6 until it actually tries, so it will usually still ask for "AAAA" records so that it can get an address and try it.  Also, whether or not the DNS server has an IPv6 connection is irrelevant - if AAAA records exist it is required to reply with them, and if they don't its required to reply saying they don't; the LEA DNS server was doing neither.

The LEA were informed that their DNS server was breaking when queried for the AAAA records, so they replied saying they had pinged a few things and used done some nslookups (presumably only for the A record!) and they couldn't see a problem.

We ran more tests - looking up the A records worked fine (successful response in 9 milliseconds), looking up the AAAA records failed (failure response in 17 seconds) and looking up the AAAA records through a different DNS server (successful response in 23 milliseconds).  We even gave them transcripts of the tests so that they would know exactly how we tested it and would be able to reproduce it themselves.

The LEA responded with words to the effect of "well no one else has reported this problem", so we ran the same tests from a different school within the same area and demonstrated that they had the same issues.  Again, we sent transcripts of the tests to the LEA (* see footnote).

The LEA then started asking whether the school was using a transparent proxy and what the school's internal domain name is - none of this is relevant to the problem being reported.  We weren't reporting problems with the transparent proxy, or any of the school's internal servers, we were specifically reporting a problem with the LEA's DNS server.

We did some further investigation and got more detail on which DNS lookups were failing, sent this to the LEA together with more transcripts of tests and an offer to work with them to help.  Rather than asking for our help, the LEA closed the ticket as "resolved", but provided no explanation.  We reran the tests, sent them another transcript demonstrating that nothing had been fixed.

The problem was originally reported at the start of the summer holidays.  Two months later the new term started - still the problems weren't fixed, still the LEA hadn't taken us up on our offers to help them (for free!) and now it transpires a lot more domains are affected than we originally investigated.  Its causing really serious problems for the school, so the school started banging heads together and someone from the LEA actually called us.  I explain the problem yet again and he goes off saying he needs to look up some more information.

Then they start talking about transparent proxying again, and again I have to point out that we are reporting a problem with the DNS server and that this has nothing to do with the transparent proxy.  Again, I send them an email describing the problem, providing transcripts of tests, etc.  LEA techie tells the school that I didn't send any information and that I just forwarded his email back to him - I'm a bit stunned about this since it means that (1) he has never seen an email with inline comments before, and (2) he didn't read past the first line of the email.  So the email gets resent to him.

The LEA reply with some screenshots of some tests they have done which they say show that there's no problem:

• They logged into the leased line router and queried the network interface statistics that show no line errors.
• They pinged a few machines.
• They tracerouted to somewhere.

i.e. they didn't test the thing we actually reported being faulty.

The LEA suggests that this is happening because they don't provide IPv6 connectivity (as mentioned above, whether or not IPv6 is available doesn't actually change anything from a DNS perspective - clients still look up AAAA records and DNS servers are still expected to reply).

Now they say they've poked lots of holes in their firewall because they "have no information on what port AAAA records would be using" (errm, 53, the same as every other DNS request in the world?!) and could we retest - unsurprisingly its still broken.

As far as I can see:

1. They haven't actually run the tests (which we've told them how to run!) to try and reproduce the problem.  They've tested a few other things that were never a problem to begin with.
2. They don't understand enough about DNS (which is an extremely fundamental internet protocol) to diagnose the issues - they seem to have entered a "change something at random and see if it fixes it" phase instead of trying to get to the root of the problem.
3. They are completely out of their depth - if they want to run a reliable WAN, they need someone wuo is actually qualified to administer a network.  That means someone who understands how to reproduce problems, use debugging tools such as WireShark, etc.
4. They haven't handled this in a timely way at all - they had the whole of summer to investigate, and didn't actually start looking at anything in earnest until after the start of term.

I have spent literally hours on this problem, mostly repeating the same explanations and tests over and over (although strictly speaking this isn't "our problem", diagnosing and liaising with the LEA is something we're handling as part of the customer's advanced support contract, so we're not really being paid by the LEA for this level of hand-holding).  I honestly can't see them resolving this problem until they reproduce it themselves and do some proper diagnostics.

Footnote

As mentioned, part of the LEA's defense is basically "no one else has reported a problem" - now, not looking into a problem because it isn't affecting many people is a pretty crumby attitude to begin with, but there are reasons why some people would be affected and some not.

Fundamentally, how services, such as DNS, are expected to behave are defined by standards.  These boil down to rules like "when a client sends a request like this, the server must send a response like that".  Software that relies on these services is written to expect them to follow the rules laid out by the standards, and there is no standard set of rules saying how to handle a service that is breaking the rules - it is extremely difficult to draw up a standard explaining how to deal with something breaking the standards, simply because there are so many ways the standards could be broken!

So you may have two different pieces of software that do basically the same job, call them A and B.  In an environment where everything is sticking to the rules, they both work equally well since this behaviour is standardised.  However, if some service isn't sticking to the standards then they will often handle this differently - maybe software A still works fine, but software B breaks.  In a different situation the roles may be reversed, with software B working ok.

So its possible for a real problem, such as this, to go unreported simply because a lot of people happen to be using software that, by chance, isn't badly affected by the broken service.  Its also possible for problems to go unreported because people write off the problem as "software A is broken" and so don't report the issue to the operator of the broken server.

Counter-Terrorism and Security Act 2015

Firstly, lets get a disclaimer out of the way: I am not a lawyer, none of this constitutes legal advice, etc.  I am also of the opinion that the threat of terrorism is minuscule and that if all the effort the government puts into anti-terrorism were instead put into road safety or health care, a lot more lives would be saved, but anyway...

It has come to my attention that one of our competitors has been engaging in a bit of scare mongering in an effort to sell their product.  The following email from them has been going around a number of schools:

As of July 2015, schools across the UK are subject to a duty under the Counter-Terrorism and Security Act in which they are required to have "due regard to the need to prevent people from being drawn into terrorism". This duty is known as The Prevent Duty.

[We have] been working with various governments around the world for over a decade, developing solutions to help schools and colleges protect their students from potentially harmful sites and information. All of our solutions have been developed purely for education, based on feedback from IT Staff, teachers and IT Professionals to ensure that they have the tools they need to prevent and report on dangerous activity.

Our Web Filter is a fully customisable, cloud based solution that allows granular filtering and reporting with great ease. The Web Filter includes Suspicious Search Engine Queries, Internet Lockouts and real time updates.

Click here to learn more or contact us directly.

The Counter-Terrorism and Security Act 2015 is a pretty long and convoluted bit of legislation, but thankfully there's a Prevent Duty Guidance document, which is significantly easier to read and provides more specific guidance on what institutions are actually expected to do.  The following is a brief analysis of the parts of the guidance that relate to ICT operations within schools - there are numerous non-ICT responsibilities listed in the guidance which I won't cover here.

The guidance immediately makes clear that no "new functions" are conferred upon a school (paragraph 4):  You don't have to do anything you weren't already doing, you're just expected to place an appropriate amount of weight on preventing people from being drawn into terrorism.

There is no specific mention of monitoring internet activity, although there are several suggestions that internet filtering should be considered (paragraphs 45, 71 and 97).

A passing mention to having "effective IT policies in place which ensure that [signs of radicalisation] can be recognised and responded to appropriately" is made (paragraph 79), but there are no specific policies suggested.  Institutions must have clear policies relating to the use of equipment, especially with regards to legitimate research into terrorism/counter-terrorism as part of learning (paragraphs 97-98).

Institutions should develop an action plan to set out any actions that they will take to mitigate the risks (paragraph 90).

In short, most schools already have an existing filtering solution and robust policies regarding the use of equipment, and that is really all that is required.  (And if they don't already have this, they should, for many reasons besides this legislation.)  There certainly seems to be no requirement to replace existing systems, unless they have unusually poor capabilities.

Comparing our Iceni product with the competitor's offering, we think our customers are actually in a better position to protect their students and staff than our competitor's customers.  Not just protecting them from being drawn into terrorism, but from many other risks too; and protection is surely far more important than just meeting the minimum requirements of the legislation.

It seems that only minimal work is required to comply with the legislation - e.g. the "action plan" for risk mitigation should be drawn up and probably include information about what reports the ICT staff should be running on a regular basis, and what they should do if they find anything concerning, etc.

As always, we're very happy to work with customers to resolve any concerns, to help investigate suspicious activity and even compile data for the police.

You couldn't make it up...

I mentioned yesterday that VUR Village Hotels & Leisure Limited had settled with me for £200 + costs after unlawfully spamming.  Well you couldn't make it up - I am now in possession of a brand new spam from them, so I'm sending another preaction notice...

Another win

As I've mentioned before, spam email has become a serious problem for people - the vast majority of email traffic is spam, and unlike some of the other forms of direct marketing, the costs for the sender are minimal and are outweighed by the costs to the recipients who are having to deal with the junk emails.

What many people don't seem to be aware of is that, within the EU, spamming is unlawful.  You should be able to appeal to the Information Commissioner's Office, asking them to take action, but in reality the most they will do is write a letter to the  company in question, and most of the time not even that.  From the start of the year I've been sending legal notices to spammers asking them to pay me damages for their law breaking.  If they ignore the notice, I'm allowed to file a court claim against them (I'll write a guide on how this all works at some point soon).

I've now had several successes and this is the latest: To my knowledge, I have never been a customer of De Vere Venues, but they have been spamming me for well over 4 years - between 2011 and 2014 I received around 200 unsolicited emails from them.  They were asked on several occasions to stop, and these requests were ignored, as were subject access requests made under the Data Protection Act asking them what information they held about me.

Eventually I complained to the Information Commissioner's Office, who instructed De Vere to stop spamming me and to answer the subject access request.  De Vere admitted that they had no idea where they had acquired my details from, which tells you something about their data protection policies - even if I had been a customer, there would have been no way for them to provide evidence to support that claim since they didn't keep track of any of that information.

De Vere Venues then sold their brand and my details to VUR Village Hotels & Leisure Limited, who started spamming me using the details De Vere had been instructed to suppress.  And surprise, just like De Vere, they also ignored the two legal notices I sent them by email and the legal notice and subject access request I sent them by post.

Serving Village Hotels with court documents got their attention and they have now settled in full, out of court.  If they hadn't ignored the original emails I sent they would have saved the £25 court costs and would have had a much stronger negotiating position.

Following scripts

One of the most annoying things is contacting technical support to get a problem solved, and getting answers that have clearly come from a script, rather than being able to talk to someone who can actually put some thought into the problem.

https://secured.studentfinance.direct.gov.uk is using an expired certificate.  Here's the output from OpenSSL:

depth=2 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO SSL CA
depth=0 OU = Domain Control Validated, OU = COMODO SSL, CN = secured.studentfinance.direct.gov.uk
verify error:num=10:certificate has expired
notAfter=Apr  2 23:59:59 2015 GMT

All pretty straight forward - the certificate clearly expired at the end of April 2nd.  This causes web browsers to display very prominent security warnings (not just some easy to ignore icon somewhere - you actually have to click through the security warning and tell it that you understand that you're taking a risk).  So being the good citizens that we are, we informed them so they could fix their systems - here's the response we got back:

I would advise to clear all browser cache, cookies and temporary internet files, close the browser and reopen a new one.

How exactly does that address the problem we raised?  It doesn't does it - clearly their support department works on the principle of "don't bother looking into the problem, just tell people to clear their cookies and hopefully they'll go away".

I'm glad that we've never gone down that route with our customer support - when people phone us they don't get fobbed off with some scripted reply; they actually get to talk to someone with a lot of technical experience who can start looking into the problem immediately and provide a relevant answer.

BT's odd procedures

BT do seem to have a lot of completely bonkers ordering procedures...

A couple of our customers have ordered a bunch of leased lines from BT, but unfortunately the original orders weren't quite specific enough and the IPv4 addresses haven't been allocated quite right (we just need a bit of a routing change, some new small subnets allocating, and one of the circuits needs a bigger subnet).  Its a few weeks until these lines go in so I thought I'd preempt any problems and correct the order well in advance.

Apparently I was wrong - BT say we can't change the order without delaying the whole installation.  They advise that we should wait until the circuits are installed and then order the IP address changes and they will be done <em>on the same day</em>.

Sounds completely bonkers to me - why on earth can they do same-day changes on a live circuit (which actually requires reconfiguring it), but making changes to an order (which is just a paperwork exercise) that won't be fulfilled for weeks isn't possible without delaying it?

Oh well, no big deal, I'm just a bit stunned that configuring everything wrong and then correcting it is preferred over getting it right in the first place.

Apologetic Vision Direct

I get a lot of spam, and a good proportion of it is from legitimate British businesses.  It costs me time and effort to remove it from my mail box, time, effort and processing power to run the spam filtering software (which is only partly effective), causes me to miss important emails, breaks my concentration and is generally quite annoying.  The fact that British businesses are doing this is quite surprising given that spamming is unlawful in the EU.  In particular, The Privacy and Electronic Communications Act 2003 says that:

1. You can't send marketing mail to someone you've never done business with unless they have explicitly told you that they consent.  Consenting to a third party doesn't count (e.g. the "do you consent for our partners to send you marketing emails" tick boxes aren't allowed).
2. If you've previously done business with someone then you can send them marketing emails about "related products" so long as they either opted in, or did not opt out.  However, the ICO's guidance says that where an opt-out system is used, the opt-out mechanism must be very clear and prominent at the point where the contact details were collected.

The "legitimate business" spam I get is split between both groups.  The first group are definitely acting unlawfully; the second is not so clear-cut since I can't necessarily prove that I opted out (or that I wasn't given the option to opt out), but given that I always look for the opt-out box and try to ensure it always gets ticked whenever I hand over contact details, I think I can argue that if I didn't opt out, the option was not clear and prominent.

I've been sending notices to both groups for a while, pointing out that they are acting unlawfully, and for the most part I've had no response and continue to receive spam from them.  Where organisations have ignored these notices, I've filed complaints with the Information Commissioner's Office.  Unfortunately, the ICO's response seems to amount to writing to the offending companies and doing nothing else.

As of the start of the year, I have started sending preaction notices to legitimate British companies who's spam ends up in my inbox.  This opens the door to me being able to file a court case against them if I so wish.

VisionDirect is one company who I have made purchases from - I have bought contact lens solutions from them, and usually purchase a year's worth of solutions at a time.  As a result, they have also been spamming me for at least 2 years.  The frequency of emails varied, but it was as much as one email a week towards the end of last year, offering me things that are extremely tangentially related to contact lenses - e.g. "win 2 LUXE London Fashion Weekend tickets".

I've previously complained to VisionDirect and my complaints have been met with absolutely no response (and continued spamming).  I sent them a preaction notice on Tuesday, and less than a week later I've now had a apologetic phone call from them.  They did confirm:

1. They're aware that the ICO guidance is to use an opt-in rather than opt-out system, but have chosen to ignore it.
2. They claim there would have been an opt-out box that I hadn't ticked.  This can't have been very prominent if I missed it.
3. Their excuse for opting people in by default is so that they can send contact lens reorder reminders.  Given that they know from my order that I wouldn't need to reorder for a year, I'm not sure why they think this justified sending me 4-5 emails a month.

They have said they will review their systems.  Whether or not this actually happens or is just something they said to make me happy is another question.

Thank you VisionDirect for finally unsubscribing me from your spam list, and if you review your marketing practices so that you're not continuing to act unlawfully then that's great.  It's a shame that you ignored the previous emails and it has taken the threat of court action for you to take note.

Using an attack on freedom to attack freedom

Following the violent attack against the staff at the Charlie Hebdo offices in Paris, we've now got David Cameron calling for more powers to snoop on people.  Here's a transcript of his speech:

In our country, do we want to allow a means of communication between people that even in extremis, with a signed warrant from the home secretary personally, that we cannot read? Up until now, governments have said no, we must not.

That is why, in extremis, it has been possible to read someone’s letter, to listen to someone’s call, to mobile communications … We have a better process for safeguarding this very intrusive power than probably any other country i can think of.

But the question is are we going to allow a means of communications which it simply isn’t possible to read. My answer to that question is: no, we must not. The first duty of any government is to keep our country safe. The attacks in Paris demonstrated the scale of the threat that we face and the need to have robust powers through our intelligence and security agencies in order to keep our people safe.

The powers that I believe we need, whether on communications data, or on the content of communications, I feel very comfortable these are absolutely right for a modern, liberal democracy.

Analysis

I want to analyse what he's getting at, so I'll take the above transcript a bit at a time:

Do we want to allow a means of communication between people ... that we cannot read?

Cameron is quite specific here - do we want to allow a means of communication, not "do we want to find ways to spy on communication". It seems to me that this directly implies not allowing means of communication between people that the security services can't read - i.e. banning such mechanisms.

Up until now, governments have said no, we must not. That is why ... it has been possible to read someone’s letter, to listen to someone’s call, to mobile communications

I think it can be argued that previous governments haven't said "no, we must not" at all.  There's an important distinction between the examples he cites and the "means of communication" that he is proposing outlawing:

• A letter is sent in the clear (that is: unencrypted), and is handled by a third party (the post office).  If the security services want to read it, they can present a warrant to the post office to intercept it and simply open the letter and read it.
• A land-line telephone call is similar - it is sent in the clear and handled by the telephone company.  If the security services want to listen to it, again they can get the telephone company to intercept it for them.
• Mobile calls are slightly different - the call is encrypted as it is sent over the air.  But importantly, the telephone company decrypts it and, like a land-line call, they handle it in the clear and it can be intercepted in just the same way.

The distinction I'm getting at is that all of these means of communication are already readable, the government has just legislated access rights to the communications that are being handled by third parties.  You can send encrypted data through the post and the government won't be able to read it - no previous governments have ever legislated to stop this.

But the question is are we going to allow a means of communications which it simply isn’t possible to read. My answer to that question is: no, we must not.

He's not really added any new information here, he's just reinforcing the point that he wants to ban all means of communication that can't be spied on.

The attacks in Paris demonstrated the scale of the threat that we face and the need to have robust powers through our intelligence and security agencies in order to keep our people safe.

The attacks in Paris do indeed demonstrate the scale of the threat that we face - as horrible as the attacks were, the scale of the threat boils down to "pretty insignificant".  Lets compare it to a few other random statistics from the UK in 2012:

• 552 people were murdered
• 5,981 people aged 15 or over committed suicide
• 1,496 people died from drug misuse

And these are just a few fully preventable examples... we can also have a brief look at casualties that could certainly be reduced but aren't totally preventable: 1,754 people died from traffic accidents and a whopping 64,164 from heart disease.

In the same year, it seems there were no terrorism related deaths in the UK.  In fact, in the past 5 years there have only been 2 people killed in the UK through terrorism.

Secondly, the Paris attacks are a pretty bad excuse to extend surveillance powers - all of the attackers were already known to the security services, they just didn't have the resources to keep tabs on all suspects all of the time.  Additional surveillance powers wouldn't help here - more resources would.  But then, given the above figures, how about we instead spend some of those resources on road safety, cardiovascular treatment, drug education, mental health treatment, general police work and the hundreds of thousands of other things that are a more significant threat than terrorism?

The people at Charlie Hebdo were attacked because they were exercising their right to freedom, and here we have the British Prime Minister using them as justification for removing our freedoms.

Solutions?

So given that David Cameron wants to outlaw any communications that the government can't read, he's got a few options.  Its important to realise that encryption is used by pretty much everyone in their day-to-day lives, and most of it is actually pretty strong and not something the government can routinely break.

1. Just outlaw encryption

Lets say Cameron outlaws encrypted communications entirely.  So we can no longer do any online financial transactions - no online shopping, no online banking - doing these without using encryption would leave everyone way too vulnerable to criminals.

There are a lot of technologies, such as Chip & Pin, Oyster cards, etc. that also require encryption, but maybe he would make exceptions for that kind of stuff?

I wouldn't get much work done either - a big chunk of my work involves administering servers all over the UK from the comfort of my office.  But that involves me using encrypted connections to those servers - using unencrypted connections would leave the servers very vulnerable to being taken over by criminals.  So most of my time would be spent driving all over the country to do administration tasks that would usually just take me 5 minutes to do from my desk.  I don't really want to think of the pollution implications of this amount of driving either.

Of course, using most foreign online services wouldn't be possible since many of them require encryption and the operators wouldn't be subject to UK law requiring them to implement an encryption-free version (with all the horrible security nightmares that would entail).

So anyway, we'll assume all the law abiding citizens are not using encryption at all.  I'm sure the terrorists who think that it's ok to commit mass murder are going to comply with that law and avoid encryption too so that the authorities can spy on them.  Or maybe not...

A great idea though - maybe the authorities can spot the terrorists by seeing who is using encryption!  Well no, as it turns out, they can't - by using a combination of steganography and an old encryption method called a one time pad, anyone could send encrypted communications that are mathematically provable to be undetectable... so bang goes that theory.

In a world where encryption is outlawed, only outlaws will have encryption.

2. Outlaw strong encryption

We could have legislation that allows encryption, but only encryption that is weak enough for the security services to break.  This has most of the same problems as outlawing encryption entirely - you can be sure that if the security services can crack it, so can the criminals, and once again the outlaws themselves can continue to use strong crypto since they don't care about the law.

3. Key escrow

Probably the best option is key escrow - this is where you can continue to use encryption as normal, but everyone is legally required to hand their encryption keys over to the government.

If the keys ended up being leaked out to third parties then everyone would be screwed, of course.  It would take a massive rewrite of pretty much every piece of software that employs encryption, which seems quite unfeasible.

And again, since the terrorists don't care about the law, there's no reason why they would comply with it and hand over their keys; and as previously mentioned, there would be no way to detect that this is happening.

Summary

So there we go, Cameron has said what he wants to do, but it seems that it is all pretty much unfeasible.  Which I guess is a relief, but I suspect that this won't prevent them passing ill-conceived legislation that erodes civil liberties whilst providing no additional security.

Corollary: this is what you get when you have politicians legislating about technologies they don't understand.

The Npower saga continues

So it turns out that Npower informed the ombudsman that they had complied with the order on November 27th.  They said:

In accordance with the remedy I can confirm that we have

• Credited account for missed discounts.
• Credited disputed amount to the account to clear balance.
• Credited your account as a gesture of goodwill.
• Refunded remaining credit by cheque, you will receive this in the post in the next 7-10 working days.
• Please accept my apologies for the delay in issuing you with a correct statement and for the issues surrounding your discount.

Except I haven't received the letter or the cheque and Npower's customer services have confirmed that no cheque has been sent out.  They do claim to have complied with the resolution by crediting my (closed) Npower account, even though they have completely failed to send me an actual refund!

So as far as I can tell, Npower acknowledged the Ombudsman's orders, decided not to comply and then lied to the Ombudsman to say they had complied.  At least, I can't think of any other reason of saying "we have" complied instead of "we will" comply if they hadn't actually already done it.

So to recap, here's why I'm never going to do business with Npower again:

1. They never actually sent me any kind of contract when I opened an account with them.
2. In January 2013 they put me on the wrong tariff... of course they didn't actually tell me this immediately.
3. They set up the direct debit but didn't actually withdraw any funds from it... and again, I didn't get informed of this immediately, and hadn't noticed.
4. In July 2013 I got my first bill, which came as a shock since it was for a more expensive tariff and none of it had been paid by the direct debit they were supposed to be using.
5. Their complaints people initially said that (i) yes they had screwed up and put me on a more expensive tariff, (ii) no they weren't going to fix it or refund me because their internal systems wouldn't let them, (iii) the direct debit not being used was clearly my fault as apparently the customer is responsible for checking their bank account every month to make sure Npower have done their job correctly.  After some argument they agreed that I could cancel my direct debit until things got resolved so I didn't have to over-pay them.
6. They said they would call me back within 10 days... they didn't.
7. They started sending me nastygrams because I hadn't paid the overbilled account (for which my complaint was still being "processed").
8. They informed me that I wouldn't get my direct debit discount since I had cancelled my direct debit (with their permission).  Eventually they agreed I could have it.  I paid off the hundreds of pounds of debt that had been racked up by their failure to use the direct debit.
9. In January 2014 I found that they had "forgotten" to apply my direct debit discount.  I called them and they confirmed it had been forgotten and said they had now credited it to my account but I would need to wait 6 months for my next bill for it to be applied.
10. In May 2014 I got a demand to pay hundreds of pounds within 9 days... Yet again they had decided to simply stop using the direct debit for no reason, and waited until this had been going on for months before informing me with a "pay within 9 days or else" nastygram.
11. The still hadn't credited the direct debit discount that they said they had in January... they said it was still being "processed" - apparently it takes 6 months to "process" a credit.
12. They now swore blind that I wasn't due a direct debit discount at all, despite having previously said that they had credited it.
13. I raised a formal complaint with Npower and got a automated reply saying they would get back to me within 10 days.
14. They didn't get back to me at all, and in fact ignored all my further attempts to raise a complaint too.
15. They started sending increasingly threatening nastygrams because I refused to pay the overbilled amount.
16. The debt collection department said they couldn't do anything to stop legal action and that I would need to raise a complaint about the disputed amount.  When I pointed out I had tried to raise a complaint and had been repeatedly ignored, I was told that I should probably raise a complaint about that (!)
17. I referred the whole thing over to the Energy Ombudsman, who quickly decided on a "remedy" and ordered Npower to comply by December 24th.  Npower sent a response to the Ombudsman to say they had complied on November 27th.
18. Part of the resolution (which Npower told the Ombudsman had been actioned) was to send me a refund.  Npower have confirmed to me that this was never done.
19. Npower still claim they complied with the Ombudsman's orders on time, even though they also confirm that they never sent the refund (!).
20. They don't really seem to be in a rush to fix things - apparently I have to wait another 10 days for them to send a cheque because they can't do a bank transfer.  So assuming they actually manage to do what they say they're doing (which would surprise me, given everything else that's happened), that means I'll get my refund on January 19th, 26 days after the Ombudsman's deadline and a year after I should have received it!