Mr Bates vs The Post Office

[dmsims]

Again - banks ?

There also must be a manual backup system - what happens if a Horizon "terminal" blew up ?

CharlesElliott said:

OK - sorry, I know what centralised means, but in 2000, a truly centralised / dumb terminal system would rely on robust connectivity at all times. If you were in Anglesey and your broadband stopped working then you would have to close the post office. That wasn't acceptable.

[skwdenyer]

CharlesElliott said:

dmsims said:

Talking of sideshows

Why wouldn't they gone for a centralised system?

Not sure entirely what you mean by centralised, but this was in the early 2000s when connectivity was patchy. If it was working, terminals in branches would upload everything to a central server, but they were programmed to be able to work if connectivity was down. They would store transactions locally and then upload them centrally when connectivity was restored. They could also cope with one or more terminals in a branch going down, and sync'ing transactions between terminals locally.

Yup. If you look at the early docs, they were deploying ISDN lines to each sub post office (ISDN is, if you like, digital dial-up - a bit faster than an old modem, more reliable, but more expensive).

Back in those days there was no real broadband. Travel agents all had dedicated X25 lines (basically a phone line with an always-on modem) but they were really slow and not always reliable. And there weren’t always available in all the remote places sub post offices were located.

Leased lines were available, at extraordinary cost. You paid by the mile from the nearest city. I put in a 64k leased line to Kent back in the day - cost £15k per year, and was little faster than dial-up.

Broadband only started to be trialled in the very late 1990s. I was involved with the original DSL pilot in North London working with Video Networks (Home Choice - see https://collection.sciencemuseumgroup.org.uk/objec... for some history) but it was *years* before there was general service.

In simple terms, there was no way to reliably deploy a real-time client-server (or other centralised approach) system back in the late 1990s.

So instead they created an asynchronous system for everything that didn’t need real-time connection (card payments, cash withdrawals, etc). Once a day the system would talk to the main servers over the ISDN line, upload data and download updates.

In principle it was fine. They used a “message queue” architecture of a type that’s still used today. I don’t think the asynchronous data transfer was the cause of the problems being discussed.

There was a problem with comms between terminals in the post office. Rather than have a server and then clients, they made one terminal “master” and then expected other terminals to synchronise with the master. That seems to have caused some issues, especially if (as was common with Windows back then) the system crashed - hence I’m surprised they used Windows at all for something designed to be resilient and not needing to run regular PC software.

Edited by skwdenyer on Sunday 30th June 23:57

[skwdenyer]

CharlesElliott said:

OK - sorry, I know what centralised means, but in 2000, a truly centralised / dumb terminal system would rely on robust connectivity at all times. If you were in Anglesey and your broadband stopped working then you would have to close the post office. That wasn't acceptable.

As per my other reply, there was no broadband. This system was conceived in 1995 or so, and rolled out in the late 1990s. ISDN was the best you could have, and nobody was going to pay for an open ISDN line 24/7!

[CharlesElliott]

dmsims said:

Sure we can debate what the requirements may have been, but that wasn't what the system was designed to cope with.

[CharlesElliott]

skwdenyer said:

As per my other reply, there was no broadband. This system was conceived in 1995 or so, and rolled out in the late 1990s. ISDN was the best you could have, and nobody was going to pay for an open ISDN line 24/7!

Horizon was often connected using ISDN. And I was using Broadband as a shortcut for ISDN, which is technically not correct.

Edited by CharlesElliott on Sunday 30th June 21:18

[skwdenyer]

CharlesElliott said:

skwdenyer said:

As per my other reply, there was no broadband. This system was conceived in 1995 or so, and rolled out in the late 1990s. ISDN was the best you could have, and nobody was going to pay for an open ISDN line 24/7!

Horizon was often connected using ISDN. And I was using Broadband as a shortcut for ISDN, which is technically not correct.

Edited by CharlesElliott on Sunday 30th June 21:18

Sorry didn’t mean to come across as jumping on you! Merely for the ah-hem younger members here, it is easy to forget there was nothing we’d call broadband back then.

AIUI Horizon/Riposte didn’t dial up over ISDN (I thought it was exclusively ISDN, but may well be wrong) every time there was a transaction, and only fail over gracefully onto an asynchronous path if one was unavailable; instead it was - again AIUI - designed to phone home once a day for upload/download, and then make additional connection attempts on an as-needed basis for real-time tasks such as card authorisation.

[siremoon]

LimmerickLad said:

Im no IT person but playing devil's advocate:

Having a backdoor purposely built in kind of makes sense........it's the fact it was hidden, denied existance and access not recorded was the issue, therefore as the system worked 99.999% of the time, the system in itself was fine but it was the arsholes involved in the "cover ups" to hide its use and existance that are to blame not the system itself? I tend to see the backdoor as a bit of a built in engine management code reader that allows you to see what went wrong and then reset once the fault was fixed / rectified but happy to be corrected if I am seeing it wrongly.

I live by the motto - it isn't what you do wrong but what you do to put it right that matters.......the problem as I see it was once the 1st PM was prosecuted for something that was actually down to the faulty system, in POL & FJ's minds they couldn't admit the "back door" hence the whole lie just snowballed and, aagin IMO, became a conspiracy and a coverup at the expense of the small people in this i.e. the SPM's..those involved in this conspracy should pay a very high price IMO but I have a funny feeling there will only be 1 or 2 scapegoats and the real villains PV, JS et al and the lawyerswill get away with it scott free!

The key point imo is not that there was a backdoor in the first place but that backdoor had no audit trail on it by design. Depending on how pedantic you want to be there is no such thing as a truly immutable audit trail but not even providing one makes you question the whole design philosophy, process and mindset.

Edited by siremoon on Monday 1st July 08:24

[onetwothreefour]

siremoon said:

LimmerickLad said:

Im no IT person but playing devil's advocate:

Having a backdoor purposely built in kind of makes sense........it's the fact it was hidden, denied existance and access not recorded was the issue, therefore as the system worked 99.999% of the time, the system in itself was fine but it was the arsholes involved in the "cover ups" to hide its use and existance that are to blame not the system itself? I tend to see the backdoor as a bit of a built in engine management code reader that allows you to see what went wrong and then reset once the fault was fixed / rectified but happy to be corrected if I am seeing it wrongly.

I live by the motto - it isn't what you do wrong but what you do to put it right that matters.......the problem as I see it was once the 1st PM was prosecuted for something that was actually down to the faulty system, in POL & FJ's minds they couldn't admit the "back door" hence the whole lie just snowballed and, aagin IMO, became a conspiracy and a coverup at the expense of the small people in this i.e. the SPM's..those involved in this conspracy should pay a very high price IMO but I have a funny feeling there will only be 1 or 2 scapegoats and the real villains PV, JS et al and the lawyerswill get away with it scott free!

The key point imo is not that there was a backdoor in the first place but that backdoor had no audit trail on it by design. Depending on how pedantic you want to be there is no such thing as a truly immutable audit trail but not even providing one makes you question the whole design philosophy, process and mindset.

Edited by siremoon on Monday 1st July 08:24

Lot of violent agreeing going on here, I think! Build your widget however you like, with whatever admin tools you like, but if you're going to prosecute people relying on evidence from your widget, you have to be able to show that that evidence is what you say it is (i..e. evidence of what the defendant is accused of) and hasn't been tampered with. Exactly the same issue with the bugs: if you discover that your widget doesn't actually behave all the time as you are asserting it does, you need to let the other side know that.

If you believe GJ (and I've not seen anything to convince me he's not making this up - in particular, there was no response to the Clarke advice from within POL along the lines of "but he had been told of his duties" and "we did disclose everything we should have"), POL were essentially taking GJ's technical analysis (and implicit statements as to what was / wasn't relevant) and assuming it passed the legal test for non-disclosure without ever having made sure that he understood what that test was.

[hidetheelephants]

dmsims said:

Again - banks ?

There also must be a manual backup system - what happens if a Horizon "terminal" blew up ?

CharlesElliott said:

OK - sorry, I know what centralised means, but in 2000, a truly centralised / dumb terminal system would rely on robust connectivity at all times. If you were in Anglesey and your broadband stopped working then you would have to close the post office. That wasn't acceptable.

The argument may have been that as POL had well over 10x as many SPOs as even the most numerous bank had branches such a system wouldn't work or have a bankrupting cost in terms of reliable communication links.

[vaud]

Not convinced that is an issue. OK broadband wasn't ubiquitous at the the time but 56k dialup is enough for basic financial transactions in a batch form at end of day, and for card transactions as needed.

EPOS was doing dialup batch in the early 1990s.

[hidetheelephants]

I'm not attempting to argue it was the case, just that ICL may have approached the problem with a load of mental baggage rather than from first principles.

Edited for not making sense.

Edited by hidetheelephants on Monday 1st July 19:31

[vaud]

hidetheelephants said:

I'm not attempting to argue it was the case, just that ICL may not have approached the problem with a load of mental baggage rather than from first principles.

True.

[dmsims]

I was more to do with Infrastructure/hardware in the early 90's but remember implementing some Dove equipment for a Burroughs mainframe application

Surely the PO's system had mainframe written all over it ?

[CharlesElliott]

All this is true. Horizon had distributed transactions within a branch with more than one terminal, and then distributed transactions back to the central server. As as we know, it worked perfectly well 99.99% of the time (which is no mean feat), but when it didn't, the assumption was that the SPM was stealing the money.

[vaud]

dmsims said:

I was more to do with Infrastructure/hardware in the early 90's but remember implementing some Dove equipment for a Burroughs mainframe application

Surely the PO's system had mainframe written all over it ?

https://www.computerweekly.com/news/252496560/Fuji...

"The user interface was a touchscreen and keyboard linked to a PC under the counter which ran on the Windows NT operating system. Branch PCs were connected via ISDN to a back-end mainframe. The Fujitsu-designed Epos software on the PCs was written onto an off-the-shelf system called Riposte."

[dmsims]

Right that's why it turned into a st show then ?

Why add those extra layers ?

[vaud]

dmsims said:

Right that's why it turned into a st show then ?

Why add those extra layers ?

How do you mean by layers? A mainframe + local client with either near real time or batch upload would be the right model at the time.

[dmsims]

Seriously ?

I'm thinking of layers like the Callendar Square bug

vaud said:

dmsims said:

Right that's why it turned into a st show then ?

Why add those extra layers ?

How do you mean by layers? A mainframe + local client with either near real time or batch upload would be the right model at the time.

[Short Grain]

Just checking YouTube and can't find todays feed. Usually watch the Official feed but can't even see the Mirror feed either! Anybody else having problems? Supposed to be Tim Parker former Chair of PO, according to my schedule.

[vaud]

dmsims said:

Seriously ?
I'm thinking of layers like the Callendar Square bug

Layers aren't the issue necessarily (lots of legacy systems have lots of layers in the interest of "modernization" rather than rewriting the platform)

Sloppy governance, appalling coding, poor documentation, poor QA/testing, etc are bigger issues.

IIRC Callendar Square bug is just that - a bug.

I'm not defending Horizon or the model, but there is a difference between adding a layer to a system and a bug (which in this case was rare to trigger)