Data Retention and Investigatory Powers (DRIP) bill

In April, the European Court of Justice ruled that the routine collection of location and traffic data about phone calls, texts, emails and internet use and its retention for between six months and two years meant a very detailed picture of an individual's private life could be constructed, that this amounted to a severe incursion of privacy and it therefore contravened EU law.

In response, the British government have put forward the Data Retention and Investigatory Powers (DRIP) bill to restore their ability to snoop on everyone.  Not wanting this bill to come under too much scrutiny, it was passed by the commons after a single afternoon's debate.  51 MPs voted to do their job properly and take time to make a decision, but they were overruled by 441 votes to just push it through as quickly as possible.  Assuming the Lords agree, this will pass into law tomorrow.

Once enacted, this legislation will allow the government to require internet service providers, internet application providers and telephone companies to record and retain metadata on any members of the public, without needing a warrant.  The bill would also apply to non-UK companies like Facebook (although quite how they expect to enforce this I'm not sure).

"The police will not know who that suspect is until they come to the police’s attention, at which point they have to get historical evidence. These days, part of that historical evidence will be in data records. They have to be able to access everybody’s data records in order to find those of one particular person, because the police, no more than the rest of us, are not given powers of clairvoyance with which to anticipate who is and who is not to be a suspect. Unless or until I hear from opponents of this Bill and of data retention how the police can be expected to identify in advance those who are going to be suspected of crime, I have to say that the whole logical basis of their argument completely falls away." - Jack Straw

Traditionally, a certain amount of information was kept by an ISP/telco/whatever for normal day to day business purposes. For example, a telco may keep call metadata records for a certain amount of time for billing purposes. If the police have a suspect then I see no issue with them getting a warrant to access that data.

If someone is a suspect, then I also see no problem with the police getting a warrant to record extra information that wouldn't normally be recorded/retained. e.g. they may get a warrant to have the ISP log web requests made by a suspect, or have the telco record call audio.

However, there is a distinction between the above examples, which require the police to suspect someone and convince a judge to issue a warrant, and what the government is increasingly trying to do, which is to capture data about *everyone's* activities, specifically for law enforcement purposes, just in case they later become a suspect.  That is something that's fundamentally wrong IMHO - as someone who has committed no crime, I have a right to privacy, and that right is being violated by having data recorded and retained for law enforcement purposes.

The whole "but it won't be used unless you become a suspect" argument is flawed - once the data is there, I have no confidence that access to it will be tightly controlled. Data may be leaked by accident, on purpose (illegally), the laws regarding under what circumstances it can be accessed may not be robust enough to prevent legal access for questionable purposes, and the whole thing is subject to feature creep - the access controls may be ok now, but I can't demand the historical data be deleted if their scope expands in the future.

Once upon a time, people were  considered innocent until proved guilty, but these days it seems that everyone is treated as guilty from the start and just happen to be allowed some freedom until the authorities can figure out what crime they committed.

All three main parties are backing this attack on our privacy, but is an afternoon of debate really enough to decide to throw away everyone's freedom?

Decoding Freesat, Part 2

As I mentioned in the last post, I'm reverse engineering the Freesat transmissions in order to extract the channel numbers so I can automatically update my MythTV system to use sensible channel numbers.

I've now managed to figure out most of the important bits:  Transport stream 2315 is broadcast on 11.428GHz Horizontal, at 27.5 Mbaud, FEC 2/3 and contains a stream with PID 3002.  This stream transmits a carousel of service description tables (SDTs) and bouquet association tables (BATs).  The SDTs aren't especially interesting, so I'm ignoring that for now.

Freesat tailor their channels to groups of consumers by grouping them into bouquets - each of the four countries (England, Wales, Scotland and Northern Ireland) get three bouquets - one for standard definition receivers, one for high definition receivers and one for G2 (second generation) HD receivers, so 12 bouquets in total for the time being.  (But more about regionalisation later).

Each BAT is packetised into one or more sections, and there is one BAT for each of Freesat's bouquets.  To collect all the information, you just keep watching the carousel until you've seen all of the sections for all of the BATs.

A BAT consists of a header, zero or more "descriptors" (lumps of data that have an ID that identifies the type of data they hold) and zero or more "transport streams".  Each "transport stream" in the BAT contains zero or more descriptors that contain information relating to a DVB transport stream (i.e. satellite transponder).

The top level descriptors in the BAT include standard descriptors (bouquet name, country availability, private data specifier descriptor) and some non-standard ones:

Descriptor ID	Description
0xd4	Region table
0xd5	Unknown
0xd6	Unknown
0xd7	Unknown
0xd8	Category table

I haven't investigated any of these except for the region table.  0xd5 - 0xd7 appear to be binary data.  0xd8 looks like a list of (ID, language, category name) tuples, but I'm not sure what the category IDs are referenced by; the category names are stuff like "Entertainment", "News", "Shopping", etc.

0xd4 is the one that's of interest to me - The bouquets are geographically pretty coarse, and Freesat tailor the channels to much smaller regions.  So the south of England gets BBC One South on channel 101 whilst the East Midlands gets BBC One East Midlands on channel 101, etc.  Descriptor 0xd4 contains a list of the regions that are served by the bouquet (I don't understand why bouquets are used at all for regionalisation though, since I can't see a reason for not handling it all through this fine grained regionalisation system?)  The data in this descriptor is a bunch of variable length chunks concatenated together, with the header of each chunk containing its size so the next chunk can be found. The data format of the chunks is:

Offset (octets)	Length (bits)	Description
0	16	Region ID
2	24	Language
5	8	Length of region name
6	Variable	Region name

"Language" is a three letter text string and is always "eng" at the moment.

Now, as I mentioned above, the BAT also contains a list of transport streams, with a bunch of descriptors in each.  So looking at the descriptors within a transport stream, as well as a few standard descriptors there is descriptor ID 0xd3, which maps service IDs to channel numbers.  This contains a bunch of variable length chunks concatenated together, with the header of each chunk containing its size. The data format of the chunks is:

Offset (bytes)	Length (bits)	Description
0	16	Service ID
2	16	Unknown
4	8	Length of remainder of the chunk
5	Variable	LCN/region mappings

The "LCN/region mappings" data is a concatenated set of fixed length subchunks as follows:

Offset (bytes)	Length (bits)	Description
0	4	Unknown
0.5	12	Logical channel number
2	16	Region ID

So, we can select the appropriate bouquet (e.g. if we're using an HD receiver in England, we would choose bouquet 272, which is England HD) and pick our region (such as region 15 - "E Midlands/Central E").  In theory we can filter down the data to get a list of channel numbers and what transport ID and service ID (i.e. what channel) they are assigned to.

There are a couple of gotchas:

Firstly, one service can be assigned to multiple channel numbers.  The BBC regions are all typically available on 9xx channel numbers, but your local region is on 10x as well.

Region number 65535 appears to be a fallback or default region.  So, for example, in the England HD region, service ID 10060 (ITV 1 London) is assigned to channel 103 in regions 1, 18, 27, 31 and 38 and channel 977 in region 65535.  Region 1 is London, the other regions don't appear in the region list so I assume they are legacy IDs.  So in this example, if you're in London then ITV 1 London appears on channel 103, but if you're anywhere else it is on channel 977.

Region number 0 is a complete unknown...  See below!

Open questions

Region 0 - I can't figure it out at all.  It only seems to be used for BBC One (logical channel numbers 101 and 108), and it seems that multiple channels can end up assigned to a single channel number in region 0.

For example, looking at the Wales HD bouquet (274), BBC One London, BBC One West Midlands and BBC One South are all assigned to channel 108 in region 0.  Channel 108 isn't assigned in any other region.

Similarly, the Wales SD bouquet (258) assigns these same three channels to 101 for region 0.  We clearly can't just ignore region 0 because there's no other way to assign a channel to 101 in this case, but I can't see how set top boxes can choose between the three assigned channels.  Also, I note that the SD version of BBC One Wales (service ID 10311) isn't listed in the BAT at all - do SD Freesat receivers in Wales no longer get a regional BBC One?

Even more confusing are bouquets 272 and 280 (England HD and England G2), which seem to have BBC One Scotland HD (service ID 8901) assigned to channel 108 in region 0!

It would certainly be interesting to look at a branded Freesat decoder and see what channels appear on it; unfortunately I don't have one.

Decoding Freesat, Part 1

To watch/record TV, I use MythTV connected to a satellite receiver.  Unfortunately, MythTV's handling of channel numbering is a bit bonkers... So I've been doing a bit of reverse engineering of the data transmitted by FreeSat to try and automagically pull out the local channel numbers and update the MythTV channels database...  Information on the internet seems thin on the ground, so this is what I've figured so far:

The "FreeSat Home" transponder is the interesting one (transport stream ID 2315).  This is located at 11.428GHz Horizontal, with a symbol rate of 27500 and FEC 2/3.  PID 3002 on this transponder transmits a bouquet association table (BAT).

For each transport stream, there is an entry in the BAT, containing a descriptor tag 0xd3 and an associated lump of data.  The data is a set of variable length chunks concatenated together, with each chunk containing a length value so the offset of the next chunk can be calculated.

The chunk format appears to be:

Offset (octets)	Length (bits)	Description
0	16	Service ID
2	16	Unknown
4	8	Length of remainder of the chunk
5	4	Unknown
5 + 1 nybble	12	Local channel number
7	Variable	Unknown

I haven't been able to figure out how the channels are selected by region - for example, local channel number 101 is allocated to BBC 1 London if you're in London, BBC 1 Wales if you're in Wales, etc. but I haven't found this information in the BAT yet. Compare:

	BBC One London	BBC One West
Service ID	18 9d (6301)	18 c5 (6341)
Unknown 1	81 f9 (33273)	82 01 (33281)
Size	08 (8)	08 (8)
Unknown 2	d (13)	d (13)
LCN	3 b6 (950)	3 c5 (965)
Unknown 3	ff ff f0 6c 00 00	ff ff f0 6c 00 00

Edit: "Unknown 2", "LCN" and "unknown 3" appears to be an array mapping LCNs to regions:

	ITV 1 London	ITV 1 Granada
Service ID	27 4c (10060)	27 60 (10080)
Unknown 1	83 f3	83 f2
Size	18 (24)	0c (12)
Unknown 2	d (13)	d (13)
LCN	067 (103)	067 (103)
Region	00 01	00 07
Unknown 2	d (13)	d (13)
LCN	067 (103)	067 (103)
Region	00 12	00 27
Unknown 2	d (13)	d (13)
LCN	067 (103)	067 (103)
Region	00 1b	00 2b
Unknown 2	d (13)
LCN	067 (103)
Region	00 1f
Unknown 2	d (13)
LCN	067 (103)
Region	00 26
Unknown 2	d (13)
LCN	3d1 (977)
Region	ff ff

Descriptor 0xd4 in the BAT seems to translate the 16 bit region IDs into human readable strings (is it me, or does 16 bits sound a bit excessive for region IDs?)

Changing ISP...

BT just sent round a reminder for me to renew my annual "line rental saver"...  it seems to have gone up significantly - £159.84 (so equivalent of £13.32/month) and its kicked me into having a look at my options.  Currently I pay BT for the POTS line and then UK Free Software Network (an EntaNet reseller) get £23.70 for my internet connection.  The internet connection is a plain old ADSL2+ connection* with a /29 static IPv4 subnet and a /56 static IPv6 subnet and is currently synced at about 6.7Mbps down, 960Kbps up.

(* It's supposedly ADSL2+, but my TP Link ADSL modem won't resync properly when the noise floor increases, so I actually have to run it in G.DMT mode...  There isn't a huge difference in speed though).

So anyway, all in I'm basically paying £37.02/month for POTS and internet.  The only reason I need the POTS bit at all is because it's required for the ADSL connection - I get free evening/weekend calls from BT, but that's not really worth the cost of the line.  In fact, I think it's bonkers that BT are putting their prices up, given the increasingly wide selection of alternative providers.  SIPGate, for example, charge 1.19p/minute for geographic calls, and even my pay as you go mobile is only 3p/minute.

Unfortunately, UKFSN don't appear to do the POTS bit themselves, expecting you to use BT for that, but I have been pretty happy with them so I've been looking at other EntaNet resellers.  One that has stood out is FalconNet - they are offering FTTC internet connections (40Mbps down, 10Mbps up) for £22 and POTS for £9.50, totalling £31.50/month.  Their installation cost is £96 - given that my "line rental saver" has to be paid up front, a £96 up-front cost doesn't seem bad at all - I basically end up in credit for the first 8 months.  I fired off an email to FalconNet and they confirm that they do IPv6 and a /32 static IPv4 subnet.

This is pretty compelling: amortised over 18 months, I get a much faster internet connection for about the price I'm already paying and everything after that is a saving; and no more up-front annual fees.  I just lose the free evenings and weekends calls - FalconNet charge 1.14p/min for geographic calls, so the amount I'm saving can pay for about 8 hours of calls a month.  Although truth be told, for the sake of 0.05p/minute I'll probably just use SIPgate (or another SIP gateway).

Seems like a no brainer.

"Superfast Broadband"

Business Wales have sent around an email making local businesses aware that "superfast broadband" is being rolled out across Wales and pointing out various benefits that businesses should look into.  They seem to be talking about FTTC and FTTP connections (they don't mention how available FTTP is - I think "not very" may well be accurate).

This is all quite sensible to make businesses aware of what's going on since the less technical businesses may well not be aware of this stuff.  However, the material they've sent out looks a bit questionable to me.  For starters, they've linked to a savings calculator which works out how much you could save by shifting some of your internal equipment out to "the cloud".  One of the provisos they give is "If your organisation hosts more than 1 server it will use it purely for file, print or authentication purposes" but they don't mention anything about what services you're using if you only have a single server - I'm pretty sure their pricing doesn't cover a lot of the services we run on our servers - i.e. one of our servers runs an Asterisk phone exchange, Subversion/Trac revision control, support ticketting system, a name server, email service, etc.  We also have several other servers doing different things, but in principle we could consolidate a lot more stuff onto a single server if we wanted to.

But that proviso also listed some slightly insane stuff - why would you want a print server to be in "the cloud"?  Print jobs can be *huge*, so sending them up over the internet to a print server, only to be pulled down again to the printer seems nuts to me.  Also, your printer is going to need to be connected to the network in order to talk to the print server, so is there actually much benefit in having the workstations talk to a separate print server instead of just sending jobs directly to the printer?

The other thing that caught my eye was that they are saying that servers hosted in the cloud are free.  They are suitably vague about where they're these free services come from - Google Apps, for example, is £33/user/year - certainly not free.

They do talk about the SLA provided by the cloud services, quoting figures like 99.9% uptime, but what about the reliability of the internet connection itself.  This certainly shouldn't stop people from considering cloud services, but you do need to consider that if your business is wholly reliant on a fast internet connection for everything then you're going to have serious problems if that connection is down for a long time.  BT have been known to take several days to repair faults, and whilst you can pay for an SLA that "guarantees" a fast fix, the fact remains that BT frequently miss the guaranteed fix times and the compensation is a pittance (they quote £25 compensation if they don't meet the SLA).  A small office can get by with a 3G connection for a few days if they are just using the internet for some web surfing and email; but this isn't going to work if you're expecting to shift move many gigabytes of files and print jobs between your network and a cloud service.

There's also some blurb about how to choose an ISP, but it seems to muddle lots of bits and pieces together here.  They talk about the ISP hosting your domain, email, website but don't (in my opinion) make it clear that these things might already by hosted elsewhere.  There is no discussion about why hosting these things with the ISP may, in fact, not be a great idea (e.g. tieing your web hosting into your internet service may make changing either one of them in the future quite tricky).

Finally, there's absolutely no mention about IPv6 connectivity.  This is going to become a big deal - the only part of the world that has IPv4 addresses left is Africa, and they are forecast to run out in around 5 years; for everyone else, there are no spare IPv4 addresses - any time an address is needed it has to be repurposed from another service.  There are a few ways of dealing with this problem, but fundamentally everyone needs to migrate onto IPv6.

Pretty much every ISP is going to need to implement either CGNAT or NAT64 as a stop-gap measure to ensure IPv4-only services remain accessible.  However, these technologies are bodges, each with their own list of problems and the only real solution is going to be to move everything to IPv6.

Some ISPs, such as A&A and EntaNet have been providing IPv6 connectivity for years and if you're on one of these and have a properly configured router you don't need to worry about it at all.

Some other ISPs have said they won't be rolling out IPv6 any time soon - for example, PlusNet have said they are going to use CGNAT instead.  As mentioned above, CGNAT isn't really a solution, it's a stop-gap measure with a whole host of problems, so prolonging the use of the stop-gap by not rolling out IPv6 seems extremely questionable.

Most ISPs have remained completely quiet on the issue, and anyone using these ISPs should certainly be concerned since they have provided no indication about how well they have been planning for the future.  To be clear: all ISPs have had almost 20 years to plan and roll out IPv6 infrastructure, so there isn't really any excuse for this not to be well under way.

IPv4 running out

Well, it was bound to happen sooner or later - Microsoft say they have
run out of US IPv4 addresses for their Azure service and are now starting to assign non-US addresses to US services.

This follows LACNIC (the regional registry responsible for Latin America and the Caribbean) announcing that their Gradual Exhaustion and New Entrants policies had come into effect on Monday as they hit their last /10 (around 4 million IPv4 addresses).  This essentially means that the remaining addresses are going to be allocated on an extremely strict policy, which to all intents and purposes means there are no more addresses to be had from LACNIC.

APNIC (Asia Pacific) and RIPE (Europe) have been running under their respective end-game policies for some time now, and ARIN (North America) are also operating under their "phase 4" policy, although this policy seems very vague compared to those of the other RIRs.  So the only region where IPv4 addresses are still being handed out is AFRINIC (Africa).

AFRINIC have a relatively large number of addresses left and don't really seem to burn through them very fast, probably owing to relatively slow technological development within the region - they have about 3.14 /8s (about 53 million addresses) and are projected to run out some time around the middle of 2019 with their current rate of consumption.  It will be interesting to see if the AFRINIC consumption now increases as global businesses, such as Microsoft, start requesting assignments from AFRINIC to be used in other regions.  Indeed, I wonder if the history of the applicant will count against them - will AFRINIC refuse to hand addresses to Microsoft, for example, because they now have a history of using large numbers of non-US addresses on US services?

I note that The Register has made a few silly comments in their article on this latest news that probably need some clarification:

Firstly they ask "why Azure is relying on Ipv4".  Microsoft's blog post didn't mention anything about the internals of Azure - it didn't indicate that the actual infrastructure was affected at all.  Indeed, the internals are probably running on IPv6, or private IPv4 addresses, so wouldn't be affected by the IPv4 address shortage.  However, The Register has failed to figure out that the people hosting services on Azure want the general public to be able to access them, and with the majority of ISPs still dragging their feet to roll out IPv6 to their customers there's no alternative but to use IPv4 on the server end too.

The other question they pose is "how Redmond let itself run out of IPv4 addresses" when they were "buying IPv4".  Now to be clear: IPv4 addresses have never been a purchasable commodity.  Addresses were assigned by the regional registrars according to strict criteria - at no point could Microsoft have gone to a registrar and said "can we have enough addresses to last us 10 years please" - the registrar would've told them to get lost (RIPE's policy was for LIRs to only get 6 months' worth at a time, for example).  Now, MS did, in fact, buy a bunch of IP addresses from the bankrupt Nortel in 2011, and there was some surprise that the registry didn't just demand that they be returned since strictly speaking this kind of sale doesn't seem to be allowed - the terms of most registries is that you return addresses to the registry when you stop using them.  Arguably a company that is selling off thousands of addresses is no longer using them, so they should've probably been returned rather than being sold.  In any case, in the general case, IPv4 addresses have never been for sale, so its hard to fault Microsoft for not buying up the world's supply years ago.

What is clear, however, is that everyone, especially ISPs, seriously needs to pull their finger out and roll out IPv6 soon.  Its long past the point where we can expect a slow trouble-free migration, we're well into a period where there's going to be some major disruption as people who have been dragging their feet suddenly spot the sky falling in on them.

Npower, yet again

So after Npower's previous screwup, they've done it again.

To recap, In January 2013 I switched my gas and electricity to Npower.  They set up the direct debit and everything, but sent me no paperwork at all until seven months later when I got my first bill... and it was enormous.  Coz as it turned out they had put me on completely the wrong tariff and hadn't actually bothered to charge the direct debit (I hadn't noticed).  It was merry hell to sort it out and took a considerable amount of my time, with them trying to the deflect blame back to me at pretty much every step (e.g. yes they had forgotten to charge the direct debit, but apparently that was my fault, because I'm apparently responsible for checking up on them all the time to ensure they've done their job.  I do wonder what the advantage of direct debit is supposed to be over just paying manually every month if they think I have to check it every month anyway).

Anyway, it eventually got sorted out and I paid off the hundreds of pounds of debt that had amassed on my account.

In January this year, I got a letter from them saying I was overpaying and that they were reducing my DD payments by about half.  All well and good.  However, I now have a letter telling me I am hundreds of pounds in debt to them (again) and that I must pay this within 9 days of the letter date (actually, I must pay it by 26th May, but of course I didn't get the letter on time because I was away from home).  It turns out that rather than reducing the DD payments, they just plain stopped them altogether (again).

So clearly its their fault that I've ended up with a load of debt against my account again (although I'm sure they will insist that its my fault), but even if it wasn't their fault, how is it right that they first thing they've done to let me know is give me 9 days notice to pay up?  Surely if the account is set to be paid by DD but isn't actually being paid by DD then they should tell the customer immediately rather than waiting until the account is a few hundred quid in debt?

Needless to say, I've now switched energy supplier and am waiting for Npower to call me back to reach a resolution for this...  Never before have I dealt with such an utterly incompetent company.

(Edit: Having just got off the phone with them, they have applied the £88 direct debit discount that they "forgot" to deduct from my account (despite me querying it in January already) and have let me pay off the balance so I can switch supplier.  I await the revised bill that they are sending through so I can check it's actually accurate this time.  I do wonder how much money they make off "forgetting" things like DD discounts and hoping that people don't notice.  Maybe I'm just cynical, but I feel like there isn't much incentive for them to fix their systems when the billing errors are making them money.)