10G networking - tips?
Discussion
I posted a while back about 2.5G networking. I've since found that consumer-grade 2.5G is giving me oddness when doing multi-terabyte jobs. Perfectly fine for "normal" consumer use, but it's causing me a real headache. I've decided to cut my losses and just go with 10G optical.
I've bought:
A Mikrotik 10G switch (SFP+ cages for each 10G link plus an RJ45 1G link to patch it into the rest of the network)
3-off 10G Chelsio T5 cards (SFP+ cages) - 1 for my workstation, 1 for each of two servers.
I need to order up optical modules (vendor lock-ins, fun!) and LC-terminated duplex fibres (due to minimal cost difference at my scale I'm probably going with OM4 in case I one day decide that 10G isn't fast enough), but before I do I need to figure out the best way to penetrate exterior walls as one of my servers is in an outbuilding.
Are these my only options or am I missing something:
1) run point-to-point from the switch to the outbuilding server - one length of fibre, no patch panels, going through two walls and my garden.
1a) have a box on the outside at each penetration points, just for protection and neatness (plus conduit to take it to ground level?)
1b) use one of those blow-out covers that you see on aerial/dish/cable installations... all the ones I've seen seem to be glued on though?!
2) have a box on the outside of the house with a patch/connection point, and another on the outside of the outbuilding. So, 3 fibres and 2 patch panels between the switch and that server.
1 removes two potential points of failure/installation error.
2 introduces an easy way to replace the middle length that will run through a flowerbed and potentially get chopped with a spade.
What would you do?
I've bought:
A Mikrotik 10G switch (SFP+ cages for each 10G link plus an RJ45 1G link to patch it into the rest of the network)
3-off 10G Chelsio T5 cards (SFP+ cages) - 1 for my workstation, 1 for each of two servers.
I need to order up optical modules (vendor lock-ins, fun!) and LC-terminated duplex fibres (due to minimal cost difference at my scale I'm probably going with OM4 in case I one day decide that 10G isn't fast enough), but before I do I need to figure out the best way to penetrate exterior walls as one of my servers is in an outbuilding.
Are these my only options or am I missing something:
1) run point-to-point from the switch to the outbuilding server - one length of fibre, no patch panels, going through two walls and my garden.
1a) have a box on the outside at each penetration points, just for protection and neatness (plus conduit to take it to ground level?)
1b) use one of those blow-out covers that you see on aerial/dish/cable installations... all the ones I've seen seem to be glued on though?!
2) have a box on the outside of the house with a patch/connection point, and another on the outside of the outbuilding. So, 3 fibres and 2 patch panels between the switch and that server.
1 removes two potential points of failure/installation error.
2 introduces an easy way to replace the middle length that will run through a flowerbed and potentially get chopped with a spade.
What would you do?
For the outside run I'd personally buy a rugged/armoured pre-terminated fibre assembly with a few spares (e.g. 4 or 8 cores) and ideally a pulling eye so you can drag it through the walls without putting undue strain on the fibre or terminations directly.
You'll end up with a bigger hole than the cable so use some foam backing rod and sealant designed for cable penetrations and you should be fine. I would definitely put some capping on the outside to protect it if possible. I have one of those galvanised metal cappings protecting Openreach's fibre where it rises into my building, it looks a bit industrial but it serves the purpose.
You can cover any distance reasonably and will probably support up to 100Gbps+ in the future
That's exactly what I'm doing in the near future to link my house to my garage where I have a leased line terminated and servers running. Conduits are already in the ground to enable this.
I would use DAC's for any of the short switch - server/workstation runs. They are dirt cheap and you don't need to mess around with individual transceivers.
You'll end up with a bigger hole than the cable so use some foam backing rod and sealant designed for cable penetrations and you should be fine. I would definitely put some capping on the outside to protect it if possible. I have one of those galvanised metal cappings protecting Openreach's fibre where it rises into my building, it looks a bit industrial but it serves the purpose.
You can cover any distance reasonably and will probably support up to 100Gbps+ in the future
That's exactly what I'm doing in the near future to link my house to my garage where I have a leased line terminated and servers running. Conduits are already in the ground to enable this.
I would use DAC's for any of the short switch - server/workstation runs. They are dirt cheap and you don't need to mess around with individual transceivers.
Edited by theboss on Thursday 6th July 18:38
Mr Pointy said:
Is there a reason you didn't go for 10G copper rather than fibre?
Yes -10G over UTP is a bit of a cludge. Much as it would have been nice to just use the Cat 6 cabling already in place, the modules are (from what I have seen) out of spec for SFP+ to the point that manufacturers who accept they exist advise to spread them out a bit throughout a switch to avoid heat issues.
- 10G DAC is only convenient for one of the connections (as it's really for in-rack use - I have a DAC twin-ax cable that I'll be testing for this connection), would involve hacking a big hole in the middle of my office floor for another, and cannot physically work for one of the connections due to distance.
theboss said:
For the outside run I'd personally buy a rugged/armoured pre-terminated fibre assembly with a few spares (e.g. 4 or 8 cores) and ideally a pulling eye so you can drag it through the walls without putting undue strain on the fibre or terminations directly.
You'll end up with a bigger hole than the cable so use some foam backing rod and sealant designed for cable penetrations and you should be fine. I would definitely put some capping on the outside to protect it if possible. I have one of those galvanised metal cappings protecting Openreach's fibre where it rises into my building, it looks a bit industrial but it serves the purpose.
You can cover any distance reasonably and will probably support up to 100Gbps+ in the future
That's exactly what I'm doing in the near future to link my house to my garage where I have a leased line terminated and servers running. Conduits are already in the ground to enable this.
I would use DAC's for any of the short switch - server/workstation runs. They are dirt cheap and you don't need to mess around with individual transceivers.
Thanks - amusingly I hadn't even considered armoured fibre despite using armoured cable for the power! Presumably I can use standard armour glands and box if I go that route?You'll end up with a bigger hole than the cable so use some foam backing rod and sealant designed for cable penetrations and you should be fine. I would definitely put some capping on the outside to protect it if possible. I have one of those galvanised metal cappings protecting Openreach's fibre where it rises into my building, it looks a bit industrial but it serves the purpose.
You can cover any distance reasonably and will probably support up to 100Gbps+ in the future
That's exactly what I'm doing in the near future to link my house to my garage where I have a leased line terminated and servers running. Conduits are already in the ground to enable this.
I would use DAC's for any of the short switch - server/workstation runs. They are dirt cheap and you don't need to mess around with individual transceivers.
Edited by theboss on Thursday 6th July 18:38
Any chance of a pic or link to the capping you used and how to cover the cable entry part? Industrial it may be, but I fear Mrs D may not be happy if I add another weatherproof box!
donkmeister said:
Yes
-10G over UTP is a bit of a cludge. Much as it would have been nice to just use the Cat 6 cabling already in place, the modules are (from what I have seen) out of spec for SFP+ to the point that manufacturers who accept they exist advise to spread them out a bit throughout a switch to avoid heat issues.
I make moderately extensive use of it for desktop applications and some server connectivity and have always bought switches with 10GBaseT ports rather than SFP+ cages for that very reason.-10G over UTP is a bit of a cludge. Much as it would have been nice to just use the Cat 6 cabling already in place, the modules are (from what I have seen) out of spec for SFP+ to the point that manufacturers who accept they exist advise to spread them out a bit throughout a switch to avoid heat issues.
xeny said:
donkmeister said:
Yes
-10G over UTP is a bit of a cludge. Much as it would have been nice to just use the Cat 6 cabling already in place, the modules are (from what I have seen) out of spec for SFP+ to the point that manufacturers who accept they exist advise to spread them out a bit throughout a switch to avoid heat issues.
I make moderately extensive use of it for desktop applications and some server connectivity and have always bought switches with 10GBaseT ports rather than SFP+ cages for that very reason.-10G over UTP is a bit of a cludge. Much as it would have been nice to just use the Cat 6 cabling already in place, the modules are (from what I have seen) out of spec for SFP+ to the point that manufacturers who accept they exist advise to spread them out a bit throughout a switch to avoid heat issues.
If you have found that elusive beast, the quiet, cool and inexpensive 10Gbase-T switch then I tip my hat.
I've got an OM3 fibre going from the house. I bought a pre-terminated, armoured cable made to the right length.
At the garage end I have one of these inside the garage but they are IP rated so can be external.
https://www.htdata.co.uk/products/product-highligh...
Use the relevant fibre coupler to terminate the external cable and then run a fibre patch lead inside to the switch. Mills sell them for £15 with a coupler and pigtails but you can get them "unloaded" to fit your choice of coupler in. I think I paid about £10 for the box and coupler.
https://www.millsltd.com/default/hellermanntyton-c...
If going external, check the version you are ordering has the rear cable port (assuming you want to go out back of the box into the building). The ones I got back in 2020 didn't have the rear port, despite it being on the datasheet at that time.
At the house end I actually used a Wiska IP rated junction box. I'd need to check which one, but it has threaded entry points, so I could screw the end of the pre-term fibre into the entry point. One thing I wish I had done here was to request shorter tails on the fibre at this end as I have the surplus carefully coiled in the box, then going through the wall and into back of a standard backbox in the house. I use an euro-module face plate to couple the external fibre to the patch going to my switch eg
https://www.euronetwork.co.uk/lc-lc-fibre-coupler-...
At the garage end I have one of these inside the garage but they are IP rated so can be external.
https://www.htdata.co.uk/products/product-highligh...
Use the relevant fibre coupler to terminate the external cable and then run a fibre patch lead inside to the switch. Mills sell them for £15 with a coupler and pigtails but you can get them "unloaded" to fit your choice of coupler in. I think I paid about £10 for the box and coupler.
https://www.millsltd.com/default/hellermanntyton-c...
If going external, check the version you are ordering has the rear cable port (assuming you want to go out back of the box into the building). The ones I got back in 2020 didn't have the rear port, despite it being on the datasheet at that time.
At the house end I actually used a Wiska IP rated junction box. I'd need to check which one, but it has threaded entry points, so I could screw the end of the pre-term fibre into the entry point. One thing I wish I had done here was to request shorter tails on the fibre at this end as I have the surplus carefully coiled in the box, then going through the wall and into back of a standard backbox in the house. I use an euro-module face plate to couple the external fibre to the patch going to my switch eg
https://www.euronetwork.co.uk/lc-lc-fibre-coupler-...
Well a bit of testing and I have 10g working between Windows 11, Debian and FreeBSD boxes. But eff me those cards run a bit hot! 10W idle/13W under load is 2 HDDs worth of heat, but it's dumping it right where you don't want it.
A lesson for anyone else considering this: I've used server cards, which have a passive low profile heatsink as they would presumably be sat in high velocity airflow in their natural environment. In a PC case, they need a bit more consideration to the cooling. The versions that have a fan on their heatsink, I now realise are probably better suited to non-server architectures (unless you are prepared to do some extra work).
The extra work for me is going to be adding a waterblock to the network card (never thought I'd be considering a water cooled network card!) for my Windows machine, and adding a fan to blow directly onto/through the heatsinks on the other two.
The other lesson is that (for a noob such as myself) the Mikrotik 4+1 is a piece o piss to work with when set to boot in Switch OS, but less so in RouterOS. It's also only capable of full 82Gb/s switching when operating as a vanilla switch so don't buy this switch if you have fancy plans.
ETA I still have all the gear co-located so haven't ordered up any armoured fibre and accessories yet, but thanks all for the advice and recommendations.
A lesson for anyone else considering this: I've used server cards, which have a passive low profile heatsink as they would presumably be sat in high velocity airflow in their natural environment. In a PC case, they need a bit more consideration to the cooling. The versions that have a fan on their heatsink, I now realise are probably better suited to non-server architectures (unless you are prepared to do some extra work).
The extra work for me is going to be adding a waterblock to the network card (never thought I'd be considering a water cooled network card!) for my Windows machine, and adding a fan to blow directly onto/through the heatsinks on the other two.
The other lesson is that (for a noob such as myself) the Mikrotik 4+1 is a piece o piss to work with when set to boot in Switch OS, but less so in RouterOS. It's also only capable of full 82Gb/s switching when operating as a vanilla switch so don't buy this switch if you have fancy plans.
ETA I still have all the gear co-located so haven't ordered up any armoured fibre and accessories yet, but thanks all for the advice and recommendations.
theboss said:
Water cooled 10Gbps networking is probably the point I just ditch the big Xeon workstation for a Mac studio!
82Gbps aggregate switching doesn't sound too shabby really... I'm sure you'll survive
Only an i9 I'm afraid, but I've got me some Cuda cores too. Good enough for the gals I go with. 82Gbps aggregate switching doesn't sound too shabby really... I'm sure you'll survive
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff