Reason For Outage report 2.04.2020

Outage window start: 2 April 2020 16:35 CEST
Outage window end: 2 April 2020 17:15 CEST

1. INCIDENT:
Both route-servers are stuck and all BGP sessions are dropped.
————————————————————

2. CAUSE:
Today a new big TIER1  network joined the Speed-IX peering community. When new BGP sessions were established by the client’s configuration mistake, a full BGP routing table was propagated to Speed-IX route servers. Because of the new member has a big network with a lot of originated IP prefixes, the max-prefix limit was configured with big values, according to peering DB configuration.
Because of two factors, such as big max-prefix limit and mistake in BGP configuration on the client’s side, both Speed-IX route-servers received a big amount of IPv4 and IPv6 prefixes and reached preconfigured memory limit. As a result, the BGP demon was crashed and all BGP sessions were lost.
————————————————————

3. SOLUTION:
– BGP sessions with new members were disabled.
– BGP demons on both route-servers were restarted
After these actions both route-server were restored.
————————————————————

4. PREVENTION:
To avoid the problem in the future we did:
– Memory on both route-serves was increased in 8x times
– Max-prefix limit was decreased for the new member

We thank you for your understanding and please report to noc@speed-ix.net in case you still have any problem with BGP peering via Speed-IX.