Friday 27 October 2023

Reason 324873 why the ICO aren't worth the £62m they are paid

The Information Commissioner's Office has an income of about £62m per year.  I receive about 1000 spam emails a year - most are from anonymous foreign actors, but a significant proportion are from registered UK businesses who are flouting the rules.  As a point of comparison, in the past year the ICO have taken "enforcement action" against just 2 organisations with respect to email marketing: Join the Triboo Limited, and Monetise Media Limited.  In both cases, the organisations were sent "enforcement notices", which they ignored, and were then fined, which probably does a fair bit to demonstrate how much "enforcement notices" are worth.

The ICO is partly funded by fines, so there's incentive to go after one or two big cases (which make them money) and ignore the thousands of small cases (which cost them money).

Anyway, without further ado, here I present reason number #324873 why the ICO are a waste of money:

Some company called Autosuggest started spamming me.  I won't give them the publicity by showing the email here, but it was badly formatted, contained my first name, my employer's name and information about the nature of my employer's business.  There was no unsubscribe link or privacy information in the email.

The email was sent to my "corporate address", so probably wasn't in breach of PECR (although the definitions are rather convoluted and could be interpreted either way).  But they've still processed my personal data, and that means they needed to comply with GDPR.

The first thing to note is that Article 14 of the GDPR requires them to have provided me with "privacy information", and they have not done so.  Its common for spammers to include a link to that info in their emails, which does fulfil the requirements so long as the emails are sent within a month of obtaining the personal data, but in this case they didn't even do that.

So, back in February, I made a Subject Access Request, which asked for various pieces of information.  The most obvious of these was "provide me with a copy of all data you hold which relates to me", but there were a few others such as asking to confirm who the data controller is, and records of processing activities.  The "right of access" is provided by Article 15, and although that particular article doesn't mention time scales, Article 12(3) does.  They are required by law to respond to my request within 1 month.

The month came and went with complete radio silence and after a second month I sent a chasing email which at least generated an automated confirmation of receipt that said it would be reviewed by a "consultant".

Needless to say, they didn't reply, so 2 months later (now 4 months since I made the original request) I raised a complaint with the ICO.  Another 2 months went by and the ICO responded, upholding my complaint and instructing Autosuggest to respond to me within 14 days.

Thank you for your email of 12 June 2023 regarding a data matter involving Autosuggest. We have considered the information available in relation to this complaint, and we are of the view that Autosuggest has not complied with their Data Protection obligations. This is because you did not receive a response to your subject access request within one calendar month and were sent unsolicited marketing communication.

We have written to Autosuggest about their information rights practices. We requested that the organisation revisit the way your SAR has been handled, provide you with all of the information you are entitled to as soon as possible or within 14 calendar days, and also stop sending you unsolicited marketing emails.

Should Autosuggest not provide you with the personal data to which you are entitled, you have the right to approach the courts for an order for your data to be released. Legal action of this nature is not something that the ICO can assist with, and we would recommend that you seek some independent legal advice before taking this step.

We will not be taking further action on this case at this time. Furthermore, please note that the organisation is not located in the U.K., so the ICO’s jurisdiction does not cover the organisation’s location. However, we hope the organisation adheres to our guidance, upholds best practises, and respects individuals privacy rights.

Please be aware that the ICO cannot award compensation, nor can we advise if it should be awarded. It is for the organisations to decide if compensation is appropriate on a case-by-case basis, or the individual may go to court to claim compensation for damage or distress caused by any organisation if they have breached the Data Protection Act.

Thank you for bringing this matter to our attention.

I note that it does point out that Autosuggest aren't based in the UK.  However, they are marketing to UK customers, using the data of UK individuals, so GDPR's extraterritorial reach does apply.  Also, I've had similar experiences with the ICO when dealing with UK companies, and a quick comparison between the number of data protection breaches, and the number of enforcement actions they have taken does quite a good job of demonstrating that they just don't enforce, even where they can.

Anyway, great - the ICO have ordered them to provide "all of the information" within 14 days, so they'll do that, right?

And indeed, they did send a response.

Thank you for contacting us.

Apologies for our late reply.

We hereby would like to inform you that regarding your personal data we have only stored your first name, last name and e-mail address in our database.

Also, we have removed your e-mail address from our marketing communication as per your wish, this has already happened previously after we have received your e-mail.

In case of any further questions, please feel free to contact us.

Hmm... not exactly comprehensive is it.  It says what types of information they hold, but not what the information actually is so no way to check if its accurate.  Also, their spam included details about my employer, but they haven't mentioned that at all.  And most of all, I asked several other questions, none of which they have answered.  But they do say I can contact them if there are any further questions, so I did that the next day and received an auto response saying my email had been received.

2 months later (now 8 months since my original request), still no response, so back to the ICO I go.  I asked them to re-open the case since Autosuggest had only provided a minimal response, which was obviously short of the "all of the information" that the ICO had told them to send.  I attached copies of the emails, etc.

Now, I'm going to look at what the ICO said in their reply in some detail:

Thank you for your email of 16 October 2023 regarding your data concern that involves Autosuggest. Please note that Autosuggest have informed the ICO that they have supplied all the information they hold on their system concerning you. They have ensured that the information provided is complete and comprehensive.

Ok... but I had sent both my Subject Access Request, and Autosuggest's response to the ICO, so the ICO can clearly see that it is not "comprehensive".  So why have they presented Autosuggest's statement as-is and not acknowledged that Autosuggest have lied to them?

If you believe that there may be other information missing or have concerns about the completeness of the information provided, it is recommended that you raise this specific concern directly with Autosuggest, giving them an opportunity to address it. By engaging with the organisation first, you can allow them time to respond and resolve any potential issues.

Right, but again, they can see from the evidence I sent them that I did raise this specific concern with Autosuggest, and they didn't reply.  So why are the ICO telling me to do something that I have already done and didn't work?

The ICO expects organisations to genuinely address the concerns raised by individuals and take appropriate action to resolve them.

Its all very well to outline these expectations, but this organisation clearly isn't meeting them so why are the ICO communicating those expectations to me instead of penalising Autosuggest for not meeting them?

Therefore, it is advisable to follow the steps outlined above before considering involvement with the ICO.

But the ICO can see, from the information I've given them, that I did already follow the steps outlined above before getting them involved, so why are they telling me this?

Should you decide to pursue the matter with Autosuggest, I recommend allowing them a reasonable timeframe like a month, to respond to your concern.

Well!  A month, you say?  Did you notice that I had given them two?

This will facilitate a constructive dialogue between you and the organisation focused on addressing your data protection rights and rectifying any potential shortcomings.

Did they really mean "constructive dialogue", or maybe they meant to say "constructive monologue" since Autosuggest won't reply to any of my communications?  Maybe the ICO should explain how a monologue can be constructive?

Please remember that our role at the ICO is to monitor compliance with data protection regulations and ensure that organisations fulfil their obligations.

But when they are faced with an organisation that is not fulfilling their obligations, what are the ICO doing to "ensure" that they do?  In my experience, except for a handful of high profile headline cases, the answer is "nothing".

We encourage you to approach Autosuggest first in order to resolve any outstanding issues you may have.

Again, why are the ICO reiterating this when they can clearly see I already "approached" Autosuggest first and got nowhere?

Thank you for your understanding, and we hope for a swift resolution to your concern with Autosuggest.

Well, whoop-di-do, thank you for all your help ICO.

Anyway, I replied to the ICO pointing all of this out and I got a rather blunt reply that just referred me back to the email above, without addressing the fact that all of the information they had provided was useless.

Oh, and they also told me they would ignore any further emails I sent the ICO regarding this... so.. yay?


So this is essentially the state of play in the UK.  There is (for now) good data protection legislation, but practically no enforcement.  So unless you are concerned about reputational damage to your company, you can completely ignore it with very little risk of penalty.  If you're a trashy little spammer, or data broker, why would you comply with the law, since your reputation is already trash anyway?

Monday 17 April 2023

ICO Failings

I have two significant examples showing why companies have basically no incentive to protect personal data.  In both cases the companies have clearly broken data protection legislation, but the ICO don't see why they should intervene:

Reel Effect Limited

This company sent a number of spam emails to me in May - July 2022.  There are a couple of things to cover first:

  1. The email address they were sending spam to belongs to an "individual subscriber", as defined by The Privacy and Electronic Communications (EC Directive) Regulations 2003 (PECR).  Since they didn't have my consent, this was unlawful (Regulation 22 of PECR).
  2. Although the email address did not contain my name, is it an address used solely by me, and if you Google for it the top link clearly identifies me.  Article 4(1) of the UK GDPR defines personal data as: ‘"any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person" (emphasis mine).  The email address is an online identifer from which I can be easily identified, so should be regarded as personal data.

I sent a subject access request to them, which Article 15 of the UK GDPR requires them to respond to within a month.  They didn't respond, so in October 2022 I made a complaint to the Information Commissioner's Office, who are the UK's data protection regulator.

Around 3 months later, the ICO concluded that there was "more work" for Reel Effect Limited to do, and wrote to them.  However, Reel Effect Limited ignored the ICO's letter, I re-reported them to the ICO in March 2023 and the ICO again wrote to them.  This finally resulted in opening up some communications from Reel Effect Limited, who essentially started making up excuses and creative interpretations of the law.  Here are some examples:

  • On why they ignored my Subject Access Request, they said that only one marketing email was sent and I didn't send the SAR until 2 months later, after they had migrated to a different domain.  This is simply not true - they sent me several spams and I sent my SAR to the sales email address listed on their website the day after the latest spam.  (They have no privacy policy on their website and list no address for data protection queries, so the sales address seemed the best option.  The spam emails stopped, so it seems pretty reasonable to assume they got the message and decided to ignore it).
  • They said that PECR does not apply to addresses that do not contain someone's name.  This is not at all true - PECR applies to all "individual subscribers".  See my comments below on the definition of an "individual subscriber", but there is no requirement for addresses to contain someone's name).
  • They then went on to claim that because the DNS host for the email address's domain is a limited company it must be a "corporate subscriber" address.  I would imagine that most "individual subscribers" contract a limited company to host their DNS, so this seems an odd position to take.
  • Then another odd claim is that since I am the director of the limited company that hosts the DNS, that makes me a "corporate subscriber".  Again, this seems like a very odd position to take - it should be of no relevance who I contract to provide DNS hosting services for my personal domains.  In this case I used a limited company for which I am a director, but I could have equally chosen another business.

So this seems fairly straight forward to me - my email address is clearly "personal data" as defined by the UK GDPR, and I am an "individual subscriber" as defined by PECR.  Imagine my surprise when the ICO decided that Reel Effect Limited has complied with their data protection obligations because they did not know they were dealing with personal data or an individual subscriber.  This absolutely boggles my mind that if you break data protection legislation, the ICO think that's ok so long as you didn't do enough due-diligence first to know you were breaking it!

This due-diligence isn't even hard - visit a website to find out what the business name or number is, then visit the Companies House website to look up the company.  My website clearly isn't a corporate website, and it contains a notice at the top pointing out that the email addresses on the domain are those of individual subscribers.  The fact of the matter is that they simply didn't do any due-diligence at all.  And its basically paid off because the ICO has now endorsed this process.

And if you receive a Subject Access Request, its probably worth answering it rather than completely ignoring it even if you don't think you're handling personal data, since it might turn out that you were wrong.

Here is the ICO's redacted response.  I've added some highlighting on the important bits:



The ICO points out that I can still take Reel Effect Limited to court, which is what I intend to do.  But I do feel that the ICO's bonkers decision does a lot to undermine my claim.

Individual Subscribers

The Privacy and Electronic Communications (EC Directive) Regulations 2003 (PECR) (amongst other things) protects "individual subscribers" from spam email.  If you're not an "individual subscriber" (e.g. the email address belongs to a limited company) then you get no protection.  But it has a rather odd and convoluted definition of what an "individual subscriber" is:

First of all, it defines an "individual" as "a living individual and includes an unincorporated body of such individuals".  This seems pretty straight forward - basically, this definition includes people, but also unincorporated businesses such as sole traders, partnerships, etc.

The definition of a "subscriber" is less straight forward: "a person who is a party to a contract with a provider of public electronic communications services for the supply of such services".  So what is a "public electronic communications service"?  Well, Section 151 of the Communications Act 2003 defines this as "any electronic communications service that is provided so as to be available for use by members of the public".

I am not a lawyer and absolutely none of this is legal advice, but as far as I can see:

  • If you (as an individual) pay a company that provides services to the public, such as your ISP, for an email address, then that is very clearly an "individual subscriber" email address.
  • But what if you self-host your email? Well, this seems much less clear - if you contract a company to host the DNS for your domain, I would think that makes you an "individual subscriber" too... Maybe you pay for a virtual machine to host your email on - that should make you an individual subscriber too, I guess. You probably also pay an ISP to provide the internet connection through which you access your email, so surely that makes you an individual subscriber?

Something that seems often missed, and which was presumably never the intent of the legislation, is that Regulation 22 of PECR simply regulates "electronic mail to individual subscribers" - the "individual subscriber" is the person that is receiving the email and the legislation doesn't appear to make reference to what email address was used to contact them.

Presumably almost everyone is an "individual subscriber" in some way or other - they have contracts with internet providers, mobile phone networks, etc.  So does this mean that this whole idea of "corporate" email addresses having no protection can be thrown out of the window?  If you spam a corporate email address and the individual it is delivered to has a contract with an ISP in a personal capacity, does that mean that Regulation 22 of PECR would apply?  I don't know, but I would love to see this tested in court!

(The legislation does not say you can spam "corporate subscribers", it says you must not spam "individual subscribers".  So if there can be some overlap between these definitions, the spammer needs to be sure that you are not an "individual" rather than checking that you are "corporate").

Either way, whether or not you can spam someone is determined by the recipient's contractual relationship with a service provider.  That might be something that the spammer can take a guess at, but it certainly isn't something that the spammer can be sure about.  The only reasonable way for a spammer to be sure that they are not breaking the law is to... not send spam!

CarParts4Less

This is a company I have bought from in the past and they have retained my details on file and then started spamming me daily.  I don't know what gets into the heads of marketing people - even if you didn't mind receiving spam, surely a spam every single day is far too much?!

I made a complaint to CarParts4Less in July 2022.  They ignored it and continued spamming me, so in October 2022 I escalated the complaint to the ICO.  The ICO responded saying they would not consider my complaint, because I had not yet complained to CarParts4Less about their failure to handle my complaint.

So the process that you have to follow when an organisation fails to comply with their data protection obligations appears to be:

  1. Complain to the organisation.
  2. Wait a month.
  3. Complain to the organisation again.
  4. Wait another month.
  5. Complain to the ICO.
  6. Wait 3 months for the ICO to possibly write to the organisation.
  7. Wait another month and hope the organisation doesn't ignore the ICO too.

Honestly, at this point I'm expecting the ICO to say that they won't do anything because I haven't yet complained to the organisation about their failure to handle my complaint about their failure to handle my complaint!

(For the record, CarParts4Less are still regularly spamming me and the ICO have done nothing).

Summary

With the way the ICO are handling data protection complaints, other than private litigation there's essentially no risk for an organisation unless they are a huge company that the ICO decides to go after to get some headlines to justify their existence.  A lot of the time the ICO will refuse to do anything, and if they decide to do something it almost always amounts to nothing more than writing to an organisation to remind them of their data protection obligations (even if they have to "remind" them multiple times!)

Between April 2022 and April 2023, the ICO fined just 7 companies for using personal data unlawfully for marketing purposes.  Over the same period, I have personally received settlements from 12 companies for the same reason - its quite a lot of work, and something the regulator should be doing instead of individuals.

Friday 10 March 2023

Anti-spam in court

As I've mentioned many times before, spamming is unlawful (except in some specific circumstances), but the Information Commissioner's Office is useless and generally doesn't do anything about it.  Thankfully, the Privacy and Electronic Communications (EC Directive) Regulations 2003 (PECR) and the UK GDPR both allow you to take civil action to protect your personal data from misuse.

Usually, when challenged, spammers admit that they broke the law and offer to pay compensation, which is generally best for everyone concerned.  But I now have two rare examples of spammers wilfully refusing to understand what they've done wrong, and taking the claim all the way to a court hearing.

This has not really gone well for these two spammers.  They have both lost, wasted everyone's time, and now have court fees to pay too.

Spammer 1

This was an eBay shop that I had made a purchase from around 5 years ago, and they subsequently started spamming me on the email address that I use exclusively for eBay.  Despite me objecting numerous times, they continued to spam me over a long period.

Since I was a customer, under PECR, they could only have lawfully spammed me if:

  1. I had given my informed, specific and unambiguous consent; or
  2. they had given me the opportunity to opt out at the time my details were collected, and in every subsequent email.

Needless to say, they didn't get my consent, and eBay provides no way for them to provide an opportunity to opt out.

Now, this is a point of weirdness: I'm extremely surprised that eBay would ever pass on my email address, and indeed the spammer says they don't.  But that address is only ever used on eBay, and the spamming started not long after my purchase...

The spammer insisted that I must have gone to their website and subscribed my address to their mailing list (obviously I didn't).  They also made the claim that maybe someone else did it since they don't use double-opt-in verification (no explanation as to who else would have access to an address that is only used on eBay, and I'm not sure how you would demonstrate consent without double-opt-in verification).  They didn't keep any records, so ultimately couldn't provide any evidence showing how they got my email address, and therefore that the way they collected it met the legal requirements.

The hearing was held over video link, because I was testing positive for CoViD at the time.  Ultimately, the judge ruled against the spammer, mainly because they had ignored my repeated objections.  However, he reduced the damages slightly as he judged my claim to be a bit too high (but the spammer has to pay court fees on top of that anyway).  He also said that I should have clicked the "unsubscribe" link in the spams, as I had an obligation to reduce my losses - nevermind that clicking links in random unsolicited emails is probably not a great idea!

So, key points:

  1. I made my claim under Articles 79 and 82 of the UK GDPR, and regulation 30 of PECR.  These are basically the bits of legislation that say "if someone breaks this law, you can take them to court".
  2. My claim was for "loss of control of personal data", which Recital 85 of the UK GDPR cites as "non-material damage".
  3. I noted that it is difficult to arrive at a monetary figure for non-material damage.
  4. I explained how I arrived at the monetary figure for damages: I considered how much I might have reasonably decided to charge if someone asked to purchase a licence to use my personal data for this purpose.
  5. The spammer had broken Regulation 22 of PECR.
  6. The spammer had broken Articles 15 and 21 of GDPR, since they ignored my Subject Access Request and objection.

Spammer 2

This was a company with whom I had never had any dealings.  They were spamming one of my business addresses, but that business is not incorporated and I operate it as a sole trader.

Since I was not a customer, they could only have lawfully spammed me if I had given my informed, specific and unambiguous consent.

Now free of CoViD, this is the first time I've actually attended court in person.  The spammer declined to attend the hearing and just provided a witness statement 7 days in advance, which meant that I had plenty of time to prepare a response to their points.  In fact, I prepared far more than I actually needed.

I had two main points to my claim:

  1. Spamming me was unlawful under PECR; and
  2. The spammer had scraped my personal data off LinkedIn and then used for a purpose which is explicitly disallowed by LinkedIn's terms and conditions.  I'm not completely sure, but I don't think this would be allowed under GDPR, since the data is being used for a purpose for which it was not originally collected.

I found this judge much better than the previous one - he wanted me to walk through step-by-step why I thought the spammer was breaking the law, so I took him through the legislation and why the exemptions didn't apply, etc. and found that this gave me a much better opportunity to explain my position.  He was pretty meticulous at asking me about definitions and looking things up as we went through the legislation.

In the end, the judge was only interested in the PECR breach.  He had decided that the spammer was clearly in the wrong, so there was no need to consider the GDPR breach since it would make no difference to the outcome of the claim.

The spammer had consistently misunderstood PECR, arguing that all "business-to-business" communications are exempt, even when I point out that sole traders are not exempt.  The spammer even cited guidance from the ICO as evidence for their position, despite that guidance specifically saying that PECR prohibits them from marketing to sole traders without their consent.  This was ultimately the spammer's undoing - they could not demonstrate any reason why their communications would have been exempt from PECR because they never addressed the reasons I had given as to why they weren't exempt.

I was asked to justify the amount I was claiming for non-material damage: this is usually a tricky one because its basically impossible to demonstrate a tangible loss, but I successfully explained that losing control of my personal data and having it used in unlawful ways is distressing.

I was also asked why I wasn't making a claim for material damage, and I explained that the time it takes to delete a small number of spams is very small so the damage is immeasurable.  However, I did make the point that the majority of email on the internet is spam sent from the likes of the defendant, and that does have a real cost, even if extremely hard to measure, since it necessitates filtering systems which occasionally throw away legitimate emails.

The judge ruled that the defendant had broken PECR, and that the £200 (plus court fees) that I was claiming was not an unreasonable amount to claim for the distress caused by their misuse of my personal data.

So, key points:

  1. I made my claim under Articles 79 and 82 of the UK GDPR, and regulation 30 of PECR.  These are basically the bits of legislation that say "if someone breaks this law, you can take them to court".
  2. My claim was for "loss of control of personal data", which Recital 85 of the UK GDPR cites as "non-material damage".
  3. I noted that it is difficult to arrive at a monetary figure for non-material damage.
  4. I explained how I arrived at the monetary figure for damages: I considered how much I might have reasonably decided to charge if someone asked to purchase a licence to use my personal data for this purpose.
  5. The spammer had broken Regulation 22 of PECR.
  6. The spammer had broken the UK GDPR since they had taken data from LinkedIn in a way that is disallowed by LinkedIn's terms and conditions, and used it for a purpose for which it was not originally collected.  (This was never considered by the judge).

Oh yes, there were also a couple of extra weird arguments from the spammer...

  • "Business contacts would expect to be contacted by B2B service providers to market and advertise their products and services" [and that therefore it's ok]... This appears to be a circular argument (this is lawful because people should expect us to act unlawfully), and it makes no sense to me at all.
  • That I'm turning a profit by taking advantage of the court... Given the time needed to navigate all the legal processes and actually attend the court (which the spammer did not), this isn't really an especially profitable use of my time!
  • That I'm up to no good because I've documented the legal processes I've gone through... Because I guess it's terrible to help people to understand the legal system that governs them?

Conclusion

I think this really demonstrates how dependant the outcome is on which judge is selected to hear your case.  I had thought that the first claim was much stronger than the first, since it involved more spam emails over a longer period of time and multiple failures to act upon my exercising my data protection rights, but in the end the judge of the first case was much more lenient and reduced my claim whilst the judge of the second case considered my claim to be reasonable and awarded the full amount.

Friday 15 July 2022

Invoicing a spammer

Maintaining control of my personal data is important, and companies who spam me usually end up on the receiving end of a nasty email exercising my rights under GDPR to find out what data they have about me, where they got it and to demand that they stop using it.

Back at the start of 2021, several companies who were spamming me all said that they had contracted another company to do their marketing, and all of them pointed at the same company.  I sent an email to the spam company consisting of:

  1. A "Notice Before Action" demanding that they pay damages for the misuse of my personal data.
  2. A "Subject Access Request" to find out what data they held and where they got it.
  3. A request to cease spamming me.
  4. A proposed contract, under which they would be allowed to send me further spam emails in exchange for a £30 charge per email.

They admitted fault and (after some more prodding) agreed to pay a £200 settlement.

Although I don't think that sending spam emails is an ethical business plan, I'm not going to name the spamming company because they have been pretty reasonable under the circumstances.

The email address that the spam was directed to was a business address.  However, I operate that business as a sole trader, so under the Privacy and Electronic Communications Regulations (EC Directive) 2003 (PECR), that makes it the address of an "individual subscriber", no different from a personal email address.  It is unlawful to send any unsolicited marketing email to an "individual subscriber" except in some very specific circumstances.

The spam company said that their systems had misidentified me as a limited company.  According to the Information Commissioner's Office, email addresses belonging to incorporated bodies such as limited companies are not those of "individual subscribers" and therefore out of the scope of PECR, so there is no prohibition on sending them spam.

The legislation is not quite so clear, and whether or not an address is that of an "individual subscriber" depends on things which are not discoverable by the sender.  Sending unsolicited email to anyone is a risk, since the sender can't know whether or not doing so would break the law.

I was assured that the problem had been fixed and would not reccur.

More recently, I've been receiving some spam from a number of different businesses, all sharing a few similarities:

  • The from addresses of the emails were all from domains which started with "ins." - for example, example@ins.example.com.
  • They all shared a number of identical non-standard email headers.

Many of the emails were really scammy looking - things like emails promoting Amazon Business coming from a variety of email addresses that don't appear to be associated with Amazon.

After some investigation, it became clear that these were from the same spam company - the one that, a year earlier, had admitted fault, paid me £200 and assured me it wouldn't happen again.

I identified 31 spam emails that they had sent, I'd already sent them a contract, and since 31 emails × £30 = £930, I invoiced them, fully expecting to end up in court.  The only thing that would make the emails they sent lawful was the contract, so I could conclude that they had accepted it.

What happened next really surprised me.  I quickly received an email from them admitting fault, pointing out that I had missed 5 emails and asking me to reissue the invoice for 36 emails × £30 = £1080.  So I did and they paid up immediately.

According to the spam company, they migrated to a new system, which introduced the error.  I can only assume that they never actually fixed the original problem of individuals being misidentified as limited companies and instead just added my address to a suppression list.  When they transferred to a new system, they presumably didn't transfer over their suppression list.

Thursday 1 April 2021

When a spammer ignores court action

It is unlawful for an organisation to send unsolicited marketing email / SMS messages to personal email addresses / telephone numbers, except in some specific circumstances:

  1. If you have been a customer of the organisation that sent the marketing (or have negotiated business with them, even if you never actually bought anything in the end);
  2. If you were given a clear opportunity to opt out of marketing communications at the time your details were originally collected; and
  3. If you were given a clear opportunity to opt out of marketing communications in every such communication.

If all of the above is true, they're allowed to send you unsolicited marketing, otherwise they aren't.  The rules are different for business addresses, and I won't get into that here - this post is specifically about spam to personal addresses.  The legislation for this is Regulation 22 of The Privacy and Electronic Communications (EC Directive) Regulations 2003 (PECR), and despite having "EC Directive" in the title, these regulations still apply post-Brexit.  Ignoring the regulations is probably also considered a misuse of personal data under the General Data Protection Regulation (GDPR)

Whenever I hand over personal data, I always make sure I opt out of marketing if there is the option, so in theory I shouldn't ever get any spam.  Of course, I still get quite a bit, because no one actually bothers to comply with the regulations.  And why should they?  About the worst thing that will happen to an organisation that ignores the regulations is a sternly worded letter from the Information Commissioner's Office.

Luckily, Regulation 30 of PECR and Article 79 of GDPR both allow civil action, so I've been sending legal notices to spammers, off and on, for a few years.  Usually this gets settled out of court, occasionally it goes as far as me filing proceedings with the small claims court (whereupon the spammer usually figures out that I'm serious and settles).

This is the first time that a spammer has completely ignored legal paperwork.  It was a bit of a learning curve for me, so I thought I'd document the process.

This all started when I made a purchase from bulkpowders.co.uk and they subsequently started sending me regular spam SMS messages.  I've got a standard email that I use in these situations, which I sent to their data protection officer.  Broadly, the email contains 3 parts:

  1. A "Notice Before Action", which outlines how they have broken the law, why I think I'm entitled to damages and how much I'm asking for.  I invite their comments and any offer of settlement.  They have 14 days to respond to this.
  2. A "Subject Access Request" under Article 15 of GDPR.  This is to find out what information they have about me and I specifically ask for information about which third parties they have shared my data with.  They have one month to respond to this part.
  3. A request to opt out of further communications, together with a contract.  The contract says that I'll charge for any subsequent communications and that sending any will be deemed as acceptance of the contract terms.

In this case, I did get a reply from Bulk Powders' DPO basically saying they were innocent because they had given me an opportunity to opt out.  I replied pointing out that the only way to opt out was to read down to the 1742nd word of their privacy policy which explains how to opt out, which certainly doesn't meet the requirements of a "simple" opt out.  They also responded to the Subject Access Request.  They did, however, continue to send spam SMS messages.

For both responses, they left it until the last possible day allowed by the deadlines before responding.  I queried whether they intended to settle damages and they said they did not, so that was that and I filed proceedings with the small claims court via Money Claim Online.  From what I could tell, Bulk Powders is a trading name of Sports Supplements Limited, so that's who I filed against.

Money Claim Online is a government website that makes it very simple to file small claims proceedings.  It costs £25, which you add onto the claim so that the defendant pays it if you win.

First of all you fill in a claim with your details, the defendant's details, an explanation of the claim and the total amount being claimed (including the £25 fee).  In my case, the total included the damages, plus the cost of each subsequent SMS message that I had set out in my original email to Bulk Powders:

The court sends a notice to the defendant and the defendant is supposed to either pay up or file a defence within 2 weeks.  Sports Supplements did neither - they completely ignored the paperwork that the court had sent them, so this was never going to end well for them.  So since they hadn't disputed the claim, I could ask the court to enter a default judgement in my favour.  This is a bit confusing because, from the claimant's perspective, the paperwork just says it is a "request for judgement" rather than saying that it is actually the judgement itself, but its all done through the Money Claim Online web site:

So now, they should just pay up, right?  Except they didn't - they again ignored the court.  So how to collect the money that I'm owed?  For larger amounts, you can employ the High Court Enforcement Officers, but for smaller amounts like I'm claiming you use bailiffs.  I was pretty unsure how to do this, and some Googling turned up lots of people saying that the bailiffs were usually slow and useless.  But it actually turned out to be dead easy and quite quick .

To send the bailiffs, I again went to the claim on the Money Claim Online web site and used the "Request Warrant" link.  This costs £77, which again gets added to the claim.  So if you manage to get paid, you don't have to pay (if the bailiffs can't extract money from the defendant, you're now £25 + £77 = £102 out of pocket).  A week later I received a letter from the court saying they had received a cheque and that it would be forwarded to me after it cleared.  It took another couple of weeks before I had a cheque from the court:

So there we go, it took a bit over 5 months from the time of my initial complaint until payment by the court.  I'm not sure exactly why they paid up in the end - did the bailiff actually show up at their door (in the middle of a pandemic) demanding money?  I originally asked for a £200 settlement, but by repeatedly ignoring the problem, Bulk Powders / Sports Supplements ended up paying out £362 (I got £260, the court got £102).  I'm not really sure what they hoped to achieve by ignoring things, especially once they started getting paperwork from the court - were they expecting me not to risk £77 to engage the bailiffs?

I still have serious concerns regarding Bulk Powders' handling of personal data, and I have made a complaint to the ICO (but they usually take 4-5 months to respond to complaints).  Not only did they send unlawful marketing emails, but I know that they have passed my details onto another company, who has also used them for marketing purposes.  Bulk Powders' privacy policy explicitly says they won't do that.  (I'm taking action against the other company at the moment, so can't really comment on that until it is resolved one way or the other).

Tuesday 9 March 2021

Making children as safe as they are offline

In a speech last week, The Information Commissioner, Elizabeth Denham said:

The internet was not designed for children, but we know the benefits of children going online. We have protections and rules for kids in the offline world – but they haven’t been translated to the online world.

— Elizabeth Denham, Information Commissioner

Neil Brown from decoded.legal posted an insightful blogpost on the perception that the internet is unregulated and dangerous, compared to the offline world.  The main thrust of the blogpost is that the offline world is not designed to be safe for unsupervised children.

The ICO's Children's Code is intended to make the internet safer for children.  This is a laudable goal, and there are certainly some parts of the code that all companies should be following to protect everyone, children and adults alike.  For example, privacy information is often quite opaque, even to adults, so the requirement to provide clear privacy information would benefit us all.

The offline world is very rarely designed for unsupervised children.  As Neil points out, even children's play areas are usually only designed to be safe for children who are under supervision.  They only prevent unsupervised children from entering by posting a sign (with complex grammar that might not be understood by children).  Washing your hands of the safety of unsupervised children by posting a similar sign on your website or app would almost certainly not be allowed under the Children's Code.

The intent of the Children's Code appears to be to make the internet safe for unsupervised children, but we don't do this in the offline world because it is usually not proportionate.

And this is the crux of the matter: its impossible to make the whole offline world safe for unsupervised children.  It would require banning essential tools, or placing huge financial burdens on vendors.  Councils would have to spend a disproportionate amount of money ensuring that unsupervised children cannot access dangerous roads.  So why do we expect to do so for the online world?

The key point is supervision: the internet should be safe for children, but we shouldn't be going to a disproportionate amount of effort to make it safe for unsupervised children.

But supervision is hard.  If your child is in the kitchen juggling knives, you'll probably notice, whereas they could be in the same room as you, doing unsafe things on their phone and you'll never notice.

Neil briefly points at a few technologies that can be used for child protection:

I use some of the measures which have come in for criticism recently — VPNs, and DNS over https — to maximise the scope of the filtering of Internet connections. More filtering, and more aggressive filtering, not less.

Indeed, I suspect that it is easier to prevent an unsupervised child from travelling to a particular place online than it is offline, if that's the path down which the responsible adult wishes to go.

— Neil Brown, decoded.legal

As Neil points out, by using a VPN you can direct your child's network traffic through system which can block access to inappropriate content and allow parents to supervise their child's online activities.  The parents can install an inspection certificate on your child's device which says "your parents' filter is allowed to decrypt and supervise this device", without allowing unauthorised decryption by others.

This means that the parents, or the child's school, can have a centralised system that they use to set parental controls and supervise the children under their care, across the whole internet.

Unfortunately, the main corporations that control online platforms have unilaterally decided that parents and schools shouldn't be allowed to supervise their children.  In 2016, Google effectively pulled the plug on inspection certificates by disabling them in all Android apps.  Facebook, Twitter and others had already disabled inspection certificates in their own apps some years before.

Without inspection certificates, fine grained filtering and supervision are off the table.  The playground has an opaque fence - you saw your child enter the playground, but you're not allowed to supervise their play.  You know the playground has a slide that is too high for a child of their age, but you're only allowed to control whether they can go into the playground, not whether or not they can go on the high slide.  Are they being bullied in the playground?  Who knows - you're not allowed to look!

There are a few technologies on the horizon which could make it even harder for parents to supervise and control their children's internet access, and historically, Google, Facebook, Twitter, et-al have imposed new privacy technologies and policies upon the public without consultation.  The problem is not the technologies themselves, but that they are unilaterally imposed on users rather than giving them the choice.  Whilst the ability to improve your own privacy is great, the decision over whether a parent can supervise their child should be made by the parents and children themselves, not by untouchable corporations.

The Children's Code does talk about parental controls and monitoring, but there is no framework or requirement to standardise them so that they can interact with a parent's centralised system.  The Children's Code's requirements will simply produce a fragmented approach.  Rather than the parent being able to set controls and supervise their child across the whole internet, they will need to log into each website and app separately.  Imagine having to log in and check separate "is my child juggling knives", "is my child playing with matches" and "is my child bullying their sibling" apps in the offline world.

Rather than demanding that all websites and apps are safe for unsupervised children, the ICO should be setting out a framework for websites and apps to interoperate with centralised systems operated by parents and schools.  They should be placing requirements on companies to consider whether their policies or technologies are detrimental to filters and supervision systems that are already in place.


Note: I am the Technical Director of Opendium, a company that specialises in network based online safety systems for UK schools.  This subject is of importance not only to parents, but to anyone or any organisation that is in a position of loco-parentis, such as schools, foster parents, etc.


Update: 12th March 2021

Neil has posted a follow-up response to this blogpost.

Firstly I'd like to say that, although Neil and I fundamentally disagree on a lot of things, it's very healthy to be having the conversation, and it underscores the fact that there is no single "one size fits all" when it comes to safeguarding children.  Everyone in a position of responsibility over children will have a different opinion on how best to protect those children, and these are the people who should be making the decisions - not governments or corporations, but parents and carers.

Also, although I certainly see technology as a very important part of online safety, I'd never advocate it as the only, or either primary, solution.  Neil is absolutely right that surveillance and supervision are not the same thing, and supervision requires carers to engage with the children and actually teach them how to be safe and to support them.  Indeed, gone are the days where schools just ticked their "online safety" box by installing a filter and letting it quietly run in the corner.  These days, schools are expected to support and teach children to be safe online.  Of course, some schools are very good whilst a few do just install a filter and treat it as a done job.  Thankfully, the inspectors are getting better at asking schools about their online safety policies.  I certainly think that there should be limits on how much carers invade children's privacy, but I also think that children can't expect absolute privacy - there's some balance to be had, and that balance isn't going to be the same for every situation.

My previous comments weren't intended as a rebuttal against Neil's original post - I saw them more as a reflection on something that I think(?) we agreed on (you should supervise children instead of trying to make the world safe for unsupervised children), but our idea of supervision obviously diverges somewhat.  I think this update is probably a rebuttal of Neil's follow up post though.

Traffic decryption

So, without further ado (quotes are from Neil's blogpost):

In most implementations, your target will never know that they are not talking directly to Facebook.

This isn't really true.  Android, for example, has a persistent notification that pops up every time you boot your device reminding you that you have authorised a third party to monitor your connection.  Its not quite as obvious in on a desktop machine, but it is certainly discoverable - clicking the padlock in Firefox clearly shows a warning.  Chrome isn't quite as good, but the information is there.  The persistent Android notification could probably be made more specific, such as telling you who you authorised to monitor your connection, rather than just that someone has been authorised.

State actors have the resources to install certificates directly in the OS's root certificate store, so there's not a lot that OS vendors can do to warn the user about that - this discussion is basically about certificates that the user has authorised themselves.

If someone has built the infrastructure to intercept and inspect your communications in this way, they can look your communications with your bank, the content of your email (and modify it!) and so on.

Entirely true, but thankfully most 5 year olds don't have bank accounts.  I think it goes without saying that how you supervise children depends on a lot of factors.  A primary factor is, of course, the child's age, and what is appropriate for a 5 year old is not appropriate for a 15 year old and certainly not appropriate for adults.  There isn't a "one size fits all" solution, so why should corporations impose one?

Walled gardens

Neil talks about using DNS whitelisting to set up a walled garden that only allows access to specific websites.  This means you to decide which websites to allow access to based entirely on their host name - the rest of the web address is encrypted.  Whilst a great idea in theory, and certainly a staple of school filtering 15 years ago, in the modern age this seems quite naive and doesn't really reflect the reality of the situation for a couple of reasons:

  1. Modern websites use resources from all over the place.  As a recent example, the government's COVID testing website uses Google's reCAPTCHA, which is hosted on www.google.com, so if you wanted to allow access to the COVID testing website, you would also need to allow access to Google web search, Google Images, Google News, Google Videos, etc.  The same is true for most websites and online services these days.  Not only does this undermine the protection of your "walled garden", but it also makes it extremely hard to actually set up the whitelist in the first place - you can't just whitelist the host name of one website, you have to figure out what other resources it needs (this usually can't be automated reliably).
  2. Harmful content is quite often stored along side safe content on the same host name.  If you're allowing access to googleusercontent.com so that various Google applications work, you're also allowing access to a lot of inappropriate content.  Since the child is probably under supervision, it may not be a big concern, but we certainly shouldn't pretend that the problem doesn't exist.

That schools are discouraged from using overly restrictive blocking policies should be an indication that a walled garden approach might do more harm than good.  Parents certainly need to make a decision as to whether its better for children to be in a very restrictive walled garden, or to be allowed to explore the internet more freely with a more dynamic system offering some protection from harmful content they might stumble across.  Again, this is a decision for the parents and carers, not for government or corporations.

Their platform, their rules

The second notion I found particularly interesting was that the private space on these companies' platforms (fixing the weaknesses in their own apps), and the operating systems they develop, should not be theirs to control, and that the decisions as to how they develop their services and products should not be theirs

Businesses, of course, have an obligation to fix security weaknesses in their own apps or platforms (although will I dispute the idea that the user making the choice to allow their communications to be decrypted by a specific party is a security weakness in the operating system).  However, where there are large sections of the population who will be negatively affected by the change, I do believe that a business has an obligation to enter into a discussion to see whether everyone can be accommodated.

Neil's opinion largely seems to be "their platform, their rules, if you don't like it go elsewhere".  But where else can users go?  There are basically 2 choices for mobile operating system:

Android phones start at about £45.  They have the aforementioned problems.

Pretty much the entire online safety sector has been asking Google for a dialogue for the last 5 years and have been roundly ignored.  I've seen numerous bug reports in the Android bug tracker, opened by online safety vendors and schools, and they have all been ignored or closed by the Android team without discussion.

In 2017, the IWF put me in contact with Katie O'Donovan, Google UK's head of Public Policy to try and open a dialogue, but Google were simply not interested in discussing the matter.  People within the Home Office have expressed similar frustrations.

So lets "go elsewhere": an old model iPhone starts from about £300 (£1000 for something more up to date).  Not everyone can afford to spend that kind of money on a phone.

As well as the cost of iPhones, I have to point at an incident that happened around 2 years ago: Apple provides a mobile device management (MDM) system, which is designed to allow businesses to manage their devices.  Parental control software was also allowed to hook into the MDM system, But then Apple changed the rules so that MDM could no long be used for parental control.  Since there was no other system that parental control software could use, there was outcry from the software vendors.  Apple ignored the vendors' concerns and banned the parental control apps from the App Store.  Only later did they reverse this decision as a result of bad press.

So there are only two mobile platforms, and they both have a history of refusing to engage with the people their decisions affect.

If what Steve means is that it should have been left to responsible adults to decide whether or not they want encryption which is MitM'able or not, they do, of course, have that choice: they are not required to let their children send traffic to Facebook or Twitter, if they don't agree with the way they operate, nor are they required to adopt the Android operating system.

The "their platform, their rules" argument could be applied anywhere: Should Facebook be absolved of any child protection obligations, because it's their platform?  Should an outdoor activity centre be absolved of health and safety obligations because it's a private location?  No, of course not - we expect private businesses, both on and offline, to adhere to various duty of care obligations.  Why should we not expect Google, Apple, Facebook, Twitter, Microsoft, etc. to have a duty of care to their users, and to undertake a proper consultation to make sure that changes they make do not undermine their users' safety?  Especially if people have been trying to make them aware of the problems for years.

The idea that we should leave private companies to do whatever they want because parents have a choice to ban their children from those platforms is ridiculous, and at odds with the government's stance with respect to Online Harms.

Surveillance companies, unilateral decisions, and consultations

Lastly, I wonder if there is a degree of double-standards at play here, in that I cannot help but wonder if the vendors of child surveillance systems operate with this degree of transparency and co-operation.

Can these vendors show that those most affected by their software — the children they surveil — were consulted?

No, almost certainly not, but I don't think this is the smoking gun of double standards that Neil wants it to be.  Certainly, as far as Opendium goes, we do not "surveil" children - we merely provide the tools for schools to safeguard the children who are under their care.  Can a CCTV camera vendor show that their customers have complied with the various laws that surround installation and operation of CCTV cameras?  Almost certainly not - in both cases, the vendor is not the company responsible for doing these things, so there is no way for them to guarantee that they have been done.

What I can say is that we do work closely with our customers, and would always advise that they must not undertake any covert monitoring.  Data protection legislation does require schools to be transparent with the children about what monitoring is being done, etc.  I'm not sure why the Information Commissioner's Office has limited the Children's Code to only online services, since much of it is equally relevant to the offline world - schools certainly should be providing clear and understandable privacy information to children.

Do children have a consequence-free option of not being subjected to these surveillance measures?

In law, the child's parents (or the people in loco-parentis) are responsible for making decisions regarding the child's safety.  Do children have a consequence-free option of not being subject to their parent's gaze while playing in the park?  Are they allowed to play in the playground without a teacher watching them?  Probably not - this is not the child's decision, because they are... a child.  It is up to their parents.

But it is certainly a discussion that a child can have with their carer.  I'm certainly aware of one case where a parent requested that their child not be monitored, and the school complied with the request (after having the parent sign a suitable waiver).  I have no idea what the legality of that situation is, given that the result might be the school failing to comply with their statutory obligations.

Anyway, that's enough for today.  As I said at the start of the update, I think these discussions are healthy and, as with politics, we're far better off having a chat about these things to try and understand the opposing point of view rather than just stand at the sidelines shouting "you're wrong". :)

Friday 28 August 2020

Adventures in Netfilter Land

We're doing some modernisation work at the moment, and part of that is moving our products to a CentOS 8 operating system. Our Opendium UTM appliance includes a firewall with a friendly web based user interface, and the back end is built upon iptables. The end user never sees the iptables bit of course - they just set up some firewall policies based on their users, groups and rule bundles:

Deep packet inspection is done in user space, but for performance reasons most of the other decision making is implemented as iptables rules. Those rules can get pretty complicated, amounting to thousands of iptables rules.

CentOS 8 has moved away from iptables, switching instead to nftables, which is claimed to improve performance, amongst other things. This shouldn't be a big deal - there's an adaption layer to allow nftables rules to be manipulated in exactly the same way as iptables rules, so in theory no need for big changes to our software.

We have plans to move more of the decision making into user space to reduce the complexity of the in-kernel rules, but we don't want to do that right now, so being able to port over the existing system with little modification is great... Except it didn't work.

iptables configurations consist of "chains", where each chain contains a list of rules. A rule is just some criteria, and an action that will be carried out if those criteria match the network traffic. Actions can be things like "ACCEPT" and "DROP" (which allow or disallow the network traffic respectively), or can be a "go to" or "jump" action that points at another chain.

So we can view iptables configurations as a directed graph, with "chains" as vertices and "go to" and "jump" rules as edges. Cycles are not allowed.

Problem 1: "Too many links"

When I tried to load the iptables rules into the Linux kernel, they were rejected with the error "Too many links". Its not a particularly helpful error, but some poking around revealed that nftables has a fixed size stack of 16, which means that your rules can only jump between chains to a maximum depth of 16 jumps.

We do have some pretty complicated rules, but they shouldn't go more than 16 jumps deep, so what's going on?

Delving into the Kernel, we find this horribleness in nft_immediate.c and similar code in nft_lookup.c:

static int nft_immediate_validate(const struct nft_ctx *ctx,
                                  const struct nft_expr *expr,
                                  const struct nft_data **d)
{
        const struct nft_immediate_expr *priv = nft_expr_priv(expr);
        struct nft_ctx *pctx = (struct nft_ctx *)ctx;
        const struct nft_data *data;
        int err;

        if (priv->dreg != NFT_REG_VERDICT)
                return 0;

        data = &priv->data;

        switch (data->verdict.code) {
        case NFT_JUMP:
        case NFT_GOTO:
                pctx->level++;
                err = nft_chain_validate(ctx, data->verdict.chain);
                if (err < 0)
                        return err;
                pctx->level--;
                break;
        default:
                break;
        }

        return 0;
}

This is part of a validation routine that happens when any new rules are added, and is responsible for the "Too many links" error if you try to add rules with a jump depth that would exhaust the 16 frame stack.

We can see that jumps and gotos are both handled the same way - pctx->level is incremented when following either a jump or a goto.  nft_chain_validate() will return an EMLINK error if the level is 16.  This seems wrong - jumps take up stack space, but gotos don't.  Looking at the rest of the code, I can't see a reason for this, so I changed it to the following (in both files), rebuilt, and that seems to solve that problem:

switch (data->verdict.code) {
case NFT_JUMP:
        pctx->level++;
        err = nft_chain_validate(ctx, data->verdict.chain);
        if (err < 0)
                return err;
        pctx->level--;
break;
case NFT_GOTO:
        err = nft_chain_validate(ctx, data->verdict.chain);
        if (err < 0)
                return err;
        break;
default:
        break;
}

There are a couple of other obvious problems with this code which I haven't tried to fix:

  1. ctx is a constant, but the const qualifier is immediately cast away.  Why do this?  Because the author likes watching the world burn?
  2. The "level" attribute of ctx (which is supposed to be a constant) is modified.  It does get restored before the function exits, but that is neglected if there's an error.  I'm guessing that the calling functions just discard the contents of ctx if there is an error, so this probably doesn't really matter, but yuckity yuck!

Problem 2: CPU Lockup

With the above fix in place I tried again and the machine crashed - it totally vanished off the network, and the console was unresponsive and eventually started showing "kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 23s!" warnings.  Great.

When a new rule set is committed, two validation routines are run by the kernel.  In nf_tables_api.c we find the nf_tables_check_loops() function, which checks the graph for cycles and rejects it if it has any; and nft_table_validate(), which calls the nft_chain_validate() stuff, mentioned above, to reject anything that would exceed the stack depth.

These functions are executed each time a change is committed.  Thankfully this is only once per commit, not once per rule - if you use the iptables-restore command, you can make multiple changes at once and just have the lot validated in one go; if you're adding rules one at a time with the iptables command then you only get to make one change at a time so the validation will be re-run for every rule you add.

Unfortunately the algorithm used by these functions is just a brute-force walk of the entire graph, potentially visiting each vertex multiple times.  The following set of rules is a pathological case (it doesn't actually do anything useful, its just a test case to demonstrate the problem):

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
A0         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain A0 (1 references)
target     prot opt source               destination         
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A1         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A1 (10 references)
target     prot opt source               destination         
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A2         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A2 (10 references)
target     prot opt source               destination         
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A3         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A3 (10 references)
target     prot opt source               destination         
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A4         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A4 (10 references)
target     prot opt source               destination         
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A5         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A5 (10 references)
target     prot opt source               destination         
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A6         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A6 (10 references)
target     prot opt source               destination         
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A7         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A7 (10 references)
target     prot opt source               destination         
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]
A8         all  --  0.0.0.0/0            0.0.0.0/0           [goto]

Chain A8 (10 references)
target     prot opt source               destination         

This is a mere 81 rules, and it takes about 5 seconds to commit them.  Adding an a single additional rule using "iptables -A INPUT -g A0" takes about another 5 seconds because it retriggers the validation, and worst of all, the CPU is locked up for the duration of the validation, leaving the machine unresponsive.

The problem gets exponentially worse, so if we add another chain and another 10 rules the machine will become unresponsive for 50 seconds.  Add another and your server will be AWOL for 8 minutes!

nft_table_validate() always walks the whole graph, nft_table_check_loops() often only walks a subsection of it, but nft_table_check_loops() is actually pretty quick even when walking over the whole graph.  nft_table_validate() is very very slow by comparison.  I couldn't immediately see why there was such a big difference in speed - they both seem to be very similar and neither really does a lot other than recursively walking across the whole graph.

What's the Solution?

The legacy iptables kernel modules still exist in CentOS 8, but the legacy user space tools aren't packaged any more.  But the legacy tools are still part of the iptables SRPM - they are even built, but then omitted from the final RPM.  It was a trivial job for me to edit the spec file, rebuild the package and continue using the legacy iptables modules rather than switching to nftables.

So in the short term, that's what we're doing.  We've got a lot of other modernisation work going on (a big overhaul of the UI and heritable policy structure) so would rather not get side tracked right now.

I've got a longer term plan to switch to using nftables natively, as well as move a lot of the decision making into user space to reduce the complexity of the rules.  We may well end up with a simple enough nftables configuration for these problems to largely go away.  Although I dislike the idea of living with a bug that could basically bring down a server if you load a particularly pathological configuration.

As mentioned above, it's not clear to me why there's a big difference in speed between nft_table_check_loops() and nft_table_validate().  There are certainly better algorithms for the validation they are doing, and even just adding some simple caching to avoid revisiting the same vertices over and over would be a big help.  I may end up revisiting this and rewriting the offending bits of the Linux kernel at some point in the future.

Bugzilla

These bugs have been submitted to the Netfilter Bugzilla at the following links: