Bank of Ireland - Banking Systems Failure - Day 12

probably not, most likely just an isolated machine fault

I was trying to hit a sacred number :smiley:

So has the real reason for the collapse of BOI IT system been revealed or are we still as clueless guarantors.

Well I heard from a friend who heard from somebody who really knows that it was a broken fibre cable and that the problem was BTs rather than BOIs. That fits with the symptoms and the long time span but its no better than hearsay I’m afraid. Anyone I’ve talked to in BOI has said that it was a network problem but my contacts are software rather than hardware. I would have thought that if this was a simple hardware problem - especially a suppliers hardware problem - that it would have been in BOIs interest to get the truth out there rather than the speculation.

This from a second hand source.
I heard it was a networking issue (switch config \ routing). My source also mentioned some other things which I am not going to repeat here as they are conjecture and more contentious than the possible root cause.

My non-technical banking source was told by his line manager that it was a “connection problem”. Nobody I’ve spoken to who works in BOI seems remotely concerned about the issue and are not aware of any higher level panic. When I point out that it seems crazy for the issue to have gone on so long they agree, but still do not suspect any conspiracy.

Neither do I for whatever it might be worth. I think the IT systems are shoddy, with no backups in key areas (core switching or backbone fibre interconnects). I doubt BT is at fault or BOI would be keen to push this in the media - although it may be a fibre problem with BOI monkey managers not wishing to admit they hadn’t considered the possibility of failure.

This should be a firing offence at an extremely high level - at least the head of IT - but doesn’t look to be the case.

Apparently this is old news (Feb 2010) but the Chip and Pin system is broken. Not necessarily relevant to BOI’s issues but thought it was interesting.

yro.slashdot.org/story/10/12/25/ … ublication

lightbluetouchpaper.org/2010 … is-broken/

Ok so a connection problem. Whatever tha means.

Considering the context was there an attack on the Banks major connectivity backbone?

Banks use private networks which cant be attacked from public networks.

You could DDOS their internet banking front end, that’s about it.

Sunday Business Post: More Problems with BOI Online banking

I was not ruling out physical attacks, sabotage for instance.

If customers can’t get at their digital records then what about internal access to digital records as well. A window to cover things up or an external sabotage. I think people are not being as broadly cynical as they should with the lack of clarity and assurances from BOI unless I missed this???

I would suggest anyone with a BOI to see this as the final straw and stop patronizing the beast which devours their soul.

There’s nothing to say the IT manager wasn’t sent to a comms room to temporarily pull a fibre cable. This could easily be blamed as a physical fault further upstream. But as even mid sized business have diverse paths (no one fibre cable failure can take the service out) this is reliant on a further failure or a bad config in the network system (not able to use the redundant path). Of course, if your provider (BT / Eircom / whoever) falls utterly on their ass their whole infrastructure (and your fibre pipes) are gone dark.

OTOH I have seen failures where the primary link goes and the second link doesn’t work… because it hasn’t worked in ages and no one checked it.

My understanding is that there were dual balanced fibre cables between data centres. There was an maintenance update which caused a hit on one path and traffic routed to the other path. When the first path came back (a matter of seconds), everything went haywire in the datacentre. The central systems shutdown and wouldn’t come back up all together, so they turned a load of stuff off and brought them up one-by-one (not without difficulty it appears).

This is third hand, but I consider the source reliable in the general terms.

zdnet.com/news/bank-says-glitch-caused-atms-to-spew-cash/6199241?tag=nl.e589
zdnet.com.au/commonwealth-atm-glitch-deliberate-339310536.htm?tag=mantle_skin;content

Am hearing that Bank of Ireland are experiencing delays in getting EFT payments to customer bank accounts today 13/12/12 due to a problem with their computer systems. Have been expecting a payment myself that as of yet has not materialised. :nin

Crap time of the year for this to happen if there really is something to this.

I’m also waiting for a payment today.

Our banking system simply isn’t fit for purpose. 5 working days to clear a business cheque? 24 hours to put through an EFT? How can that be? And then businesses not accepting cheques or drafts? It just slows everything down and makes it a real pain in the hole to do anything. Bah!

Pretty much the same way the country is run.

It does not take 24 hours to put through an EFT. They just charge an arm and a leg to do same-day EFTs.

Well, yes, it shouldn’t take more than a few milliseconds to transfer funds electronically, so my question was more along the lines of 'How the fck can they get away with it when it is so disruptive to business?'*.

*“Sir, It’s after 11am. I’m afraid it’s too late now for same day transfer. The money should arrive tomorrow morning”. * Wha?

The back end of many banks run on software systems that were designed in the 70’s and they are very slow to change. Change costs lots of money, even when the change is a roaring success. If it doesn’t go well then it can put them out of business. Most change is affected by layering on new functionality via interfaces to avoid changes to the core legacy systems, also most change is driven out of neccessity - it is neccessary for your current a/c to reflect an ATM withdrawal in near real time to prevent you overdrawing. It is not neccessary for an EFT to go through like this. Also, since most EFT’s are between different banks the systems connecting them are, in my experience, message driven (think email rather than skype) and added to multiple serial queues that are sometimes very long, driven by batches, subject to exceptions and verifications, and affect their daily liquidity to the extent that they often limit the rate of transfer outwards to match the rate inwards.

As for cheques, get real, banks want rid of them and they have no incentive to improve the speed of clearance.