Technical Blog

Aggregated Network Connections with LAG/LACP

This is a meandering post without an immediate happy outcome.

I am working on a five node ProxMox 8.1 cluster with three nodes as a Ceph cluster to host my media collection. I’m learning a bunch about Ceph and Proxmox which I’ll post about later. The media collection I am importing into Ceph is a little over 16Tb from ripping my VHS, DVD, BluRay collections of movies and tv shows. Movies end up being less than a third of that content.

So the history runs with my wife deciding a long time ago to turn off our cable subscription in favor of buying VHS Tape and DVDs or other media. We raided our local Block Buster when it went out of business. We also kept doing this even into the era of streaming service which had grown to now include content you cannot get on them. On the server side, this journey started with what I thought was an incredibly massive 4Tb USB hard drive hung off a Dell Optiplex 390 SFF (small form factor) desktop running Plex Media Server. I had a prior set of work with a DLNA server hacked into a Seagate BlackArmor NAS that I used as an initial import for the media collection. The new server quickly grew as I kept importing content since it had a DVD drive to do the ripping and a USB device for importing the VHS content. I eventually got concerned about the USB Drives reliablity and wanted to setup a better storage solution. My decision was to use Ceph rather than a NAS which is the use of lots of cheap redundant hardware vs expensive DASD or RAID hardware.

That is all to set the stage and I’ll be writing about setting up Proxmox in a later post. Lots of lessons learned and problems along the way.

Netgear GS105Ev2

Enter my pile of hardware and a problem to be solved.

I’m frugal and don’t buy new equipment so far as I can. My piles of network equipment included lots of unmanaged Netgear 5 and 8 port switches which also happened to include a Netgear GS105Ev2 which is labeled as a smart managed switch. With this in mind, I thought I could speed up my media import by using LAG or LACP to aggregate a couple of the network ports on my servers SAN (storage area network) for faster movement of data between the storage nodes. I already had a separate 1Gbps switch and separate network setup for Ceph. That was an entire lesson in itself that Ceph will not work well without a separate network. This would let me take a couple of 1Gbps ports and get them closer to 2Gbps was the theory. Ceph folks are going to yell that you need 10Gbps network for Ceph to run well. I won’t argue that 10Gbps would not be nice but adding 10Gbps network won’t happen until prices come down further.

Here enters the marketing crap from Netgear to botch that idea. Netgear searches will mention LAG and LACP being available on their Smart or Pro switches and it is even in their user manuals. However, those miss letting you know that feature is only enabled on 24 or 48 port switches. The five (5) or eight (8) port versions do not have LAG enabled and their marketing materials don’t make that very clear. It is also a closed system without SNMP support to pull metrics. The features it is missing just keep piling on. Netgear isn’t my first choice for managed switches but I was trying to use what I had which was a this little goober that isn’t good enough to be a managed switch or crappy enough to be a dumb switch. They fit a zone of stupid devices that make no sense. I’m not sure why I had it in my pile of equipment but likey somebody else learned this lesson as well and I got it handed over for free.

HP ProCurve 2810-24

Thus enters the combo of eBay and my prior experience with HP ProCurve 2500 switches. I use a ProCurve 2510 as the main work horse switch in my home network even with the majority of the ports being 100Mbps. After digging around I found that the HP ProCurve 2800 and 2900 series switches are in a very reasonable price range for less than $25 USD each. Those switches are 1Gbps ethernet for every port and four SPF (not SFP+) if I want to run cheap 1Gbps SFP DAC (Direct Attach Copper) cables. But better yet, these are real enterprise switches with SNMP, VLAN, LACP and all the other high-end network features. What they lack is a nice web interface to manage them since their Java WebUI is deprecated. So you are on the serial console command line banging out changes and saving the configurations initially. As a prior network engineer who had to do this stuff for other folks, this does not phase me. I’m just frustrated at the wasted time on that Netgear device.

So here I sit waiting on my eBay order to arrive so I can test out LAG and/or LACP port aggregation on my ProxMox cluster. I’ll probably upgrade my existing HP ProCurve 2510-24 while I’m getting this added to the mix. The VLAN support is excellent and I could use it in my home network. I’ll do a follow up on setting up the HP ProCurve 2800s and how it goes for speeding up my media collection import and any issues I hit along they way.

New Laptop update

My new to me Thinkpad T480 is doing great. The better processor and the upgraded Nvidia MX150 GPU are both getting a workout with several of the new LLM models. I maxed out the RAM, Wifi, Hard Drive, and swapped around adding all the best components/features to one laptop consolidated from several different junkers I purchased. Those components include IR Camera, WWAN, backlit keyboard, good batteries, and a nice case.

WSLv2 with Ubuntu is taking advantage of the newer CPU and RAM. The NVMe is making the disk access fly.

Processor                   Intel Core i7 Processor 8550U CPU: 1.8GHz (8Mb Cache, up to 4.00 Ghz)
Operating System            Windows 10 Pro 64bit (with option for Windows 11 Pro)
Display                     14.0" 1920x1080 with touchscreen
Graphics                    Intel's UHD Graphics 620 IGP
                            NVIDIA GeForce MX150 (2Gb VRAM)
Memory                      64GB(32+32) DDR4 2400MHz SoDIMM
Camera                      HD Camera 720p
                            IR Camera
Keyboard                    Keyboard backlit - English
Pointing Device             TrackPoint and TouchPad with Fingerprint Reader
Security Chip               Software TPM Enabled
First Hard Drive            4Tb Hard Disk Drive, NVMe
System Expansion Slots      Smart Card Reader
Battery                     ThinkPad Battery 24Wh Li-Ion Internal
                            ThinkPad Battery 72Wh Li-Ion External Rear
Wireless                    WiFi 6E M2 Wireless Card Intel AX210 NGW Bluetooth 5.3 Tri-Band 5400Mbps
Integrated Mobile Broadband Fibocom L850-GL (cellular network)

Upgrades done:

The 64Gb RAM is really nice to have for the machine learning work I tried out. The 2Gb VRAM is anemic on the GPU but enough to try a few things out and offload to regular RAM taking a performance hit.

Adding Wifi 6e is an absolute necessity and worth the $20 without a doubt.

WWAN cellular as an option is something I didn’t anticipate wanting or needing but I’ve found it as a nice feature and would look to keeping it around in future laptops. Using my cellphone Wifi sharing is nice enough but just having my laptop fully independant is worth something.

NVMe hard drive is nice for the performance improvement from SSD and not priced out of range. Another no brainer.

Backlit keyboard is not something I use extensively but nice to have around every blue moon. The touchscreen monitor fits in this category of nice to have but would not buy it.

Upgrade options:

I bought the Glass X1 Trackpad replacement on Aliexpress but found the driver support was a pain to deal with when installed and it just wasn’t the much of a difference for my usage. I ended up removing it and going back to the base trackpad without being bothered.

Changing out the Wifi or WWAN for a TPU or M2 SSD means missing out of those network features which I would rather have the cellular option and Wifi 6E than those the extra hard drive or TPU.

Summary

I would do this again in a heartbeat. This new to me laptop is great and letting me do work I could not do on my older one. The addition of the twin 4K monitors, updated GPU and expanded memory all make for a better experience. Windows 11 support gives me another couple years. I have future options like adding an eGPU via Thunderbolt 3 that were not even options. A Hackintosh option for the T480 exist where the X1 trackpad may become useful with the second laptop I have for parts.

So this was worth the effort and I would do it again.

A new to me but old laptop

I’ve been using an older Lenovo Thinkpad T460p laptop that I bought brand new May 2017 for $2210 USD with a pretty impressive discount for the time. The Thinkpad T460p released April 2016 over a year earlier from when I bought it. I over bought on the CPU/processor and bought extra power adapters and batteries upfront. It has been a great laptop but starting to show its age and I’ve been looking for a new laptop with a little more pep. Older USB is starting to be limiting and I’d like to go beyond 32Gb RAM.

Lenovo ThinkPad T460p

Processor                   Intel Core i7-6820HQ processor (8MB Cache, Up to 3.6 Ghz)
Operating System            Windows 10 Pro 64
Operating System Language   Windows 10 Pro 64 English
Display                     14.0 WQHD(2560 x 1440) IPS Non-Touch
Graphics                    NVIDIA GeForce 940MX 2GB
Memory                      32GB(16+16) DDR4 2133MHz SoDIMM
Camera                      720p HD Camera
Keyboard                    Keyboard Backlit - English
Pointing Device             UltraNav (TrackPoint and TouchPad) with Fingerprint Reader
Security Chip               Software TPM Enabled
First Hard Drive            500GB Hard Disk Drive, 7200rpm, 2.5", SATA3
System Expansion Slots      Smart Card Reader
Battery                     ThinkPad Battery 6 cell Li-Ion (72Wh) Cyl HC Rear
                            ThinkPad Battery 68+ 6.6 Ah Notebook Battery
Power Cord                  135W AC Adapter (2pin) - US
                            90W AC Adapter (Slim Tip) - power adapter - 90 Watt
Wireless                    Intel Dual Band Wireless-AC(2x2) 8260, Bluetooth Version 4.1 vPro
Integrated Mobile Broadband Integrated Mobile Broadband upgradable
Display Panel               T460p WQHD IPS AG WW PAINT
Language Pack               Publication - English
Warranty                    3 Year Depot or Carry-in

As I looked around, nothing under $4000-$5000 USD seemed to fit the bill once I added all the features and limitations. MacBook Pro and several WinTel laptops were in the running but nothing seemed like a good fit. And all the newer laptops seem to have soldered memory or limitations on future upgrades built into them. Add to that my interest in LLM and AI/ML which has me thinking about buying a used server or high-end workstation with multiple GPUs that will cost in the multiple $1000’s USD, I figure I should economize on my development laptop.

So that puts me browsing in the used market space on eBay and Craigslist.

A brief perusal brought the Lenovo Thinkpad T480 to my attention. The T480 as a January 2018 release is still an upgrade of my existing T460p that released in early 2016. I have a good bit of experience disassembling the base T-series platform as a bonus. The T480 are coming off support for large corporations and US Federal contracts. That brings the price way down in the used market.

Some key features are the T480 have an Intel i5 or i7 options in a Gen 8 CPU with and without a Nvidia GPU. It is the last generation of the T-series with removable batteries and has dual SODIMMs not soldered to the motherboard if you stay away from the slimline Ts-series.

Lenovo ThinkPad T480 https://www.ebay.com/itm/204515238921

Screen Size                 14 in
Graphics Processing Type    Dedicated Graphics
RAM Size                    32 GB
Color                       Black
SSD Capacity                256 GB
Most Suitable For           Casual Computing, Graphic Design, Workstation
GPU                         Intel's UHD Graphics 620 IGP
                            NVIDIA GeForce MX150
Processor Speed             1.80 GHz
Brand                       Lenovo
Series                      ThinkPad
Type                        Mobile Workstation
Maximum Resolution          1920 x 1080
Model                       Lenovo ThinkPad T480
Connectivity                Gigabit Ethernet, HDMI, USB 3.0, USB 3.1, USB-C, SD Card Slot,
                            Thunderbolt, SmartCard Reader
Operating System            Windows 10
Features                    10/100 LAN Card, Backlit Keyboard, Bluetooth, Built-in Microphone,
                            Built-in Webcam, Multi-Touch Trackpad, Touchscreen, Widescreen Display, Wi-Fi
Hard Drive Capacity         256 GB
Storage Type                SSD (Solid State Drive)
Processor                   Intel Core i7 Processor 8550U CPU: 1.8GHz (8Mb Cache, up to 4.00 Ghz)
Operating System            Windows 10 Pro 64
Display                     14.0" 1920x1080
Graphics                    Intel's UHD Graphics 620 IGP
                            NVIDIA GeForce MX150
Memory                      64GB(32+32) DDR4 2400MHz SoDIMM
Camera                      ???720p HD Camera
Keyboard                    Keyboard backlit - English
Pointing Device             TrackPoint and TouchPad with Fingerprint Reader
Security Chip               ???Software TPM Enabled
First Hard Drive            250GB Hard Disk Drive, SSD, 2.5", SATA3
System Expansion Slots      ???Smart Card Reader
Battery                     ???ThinkPad Battery 6 cell Li-Ion (72Wh) Cyl HC Rear
                            ???ThinkPad Battery 68+ 6.6 Ah Notebook Battery
Power Cord                  ???135W AC Adapter (2pin) - US
Wireless                    ???Intel Dual Band Wireless-AC(2x2) 8260, Bluetooth Version 4.1 vPro
Integrated Mobile Broadband ???Integrated Mobile Broadband upgradable
Display Panel               ???T460p WQHD IPS AG WW PAINT
Warranty                    ???

So I’m getting the following components used and new.

Vendor New Price Description
eBay Used $350 Lenovo ThinkPad T480 i7 32GB 256 SSD NVIDIA MX150 with power cord and docking station
eBay New $20 SSD HDD NVMe M.2 Adapter Caddy & cable 01AX994 For Lenovo ThinkPad T470 T480
Amazon New $15 USB C to HDMI with PD Charger Adapter
Amazon New $100 Elite DDR4 64GB Kit (2 x 32GB) - TED464G3200C22DC-S01
Amazon New $20 Intel AX210 IEEE 802.11ax Bluetooth 5.2 Tri Band Wi-Fi/Bluetooth Combo Adapter
Amazon New $165 Crucial P3 4TB PCIe Gen3 3D NAND NVMe M.2 SSD, up to 3500MB/s - CT4000P3SSD8
- $670 Total  

List of stuff to do to my T480

List of stuff I could do to my T480

Advantages:

Disadvantage:

Additional Options:

Buying a 10Gbps or higher network on a homelab budget

This is a project I’ve been thinking about for a long time… how to get 10gbpe+ networking in a homelab without breaking the bank.

First option is just getting some DAC Cables and dual port 10Gbpe NICs then build a point-to-point ring network. That is relatively cheap and would set me up for future switched networks. DACs could be swapped out for GBics that use copper (thermal issues) or fiber (delicate).

Next is a relatively cheap at a couple hundred dollars switch with likely a low number of SFP+ ports. This is ~$200-$500 with anywhere from 2 to 16 ports at 10Gbps. Often switches with 10Gbps advertised only have one or two ports at that speed so shop carefully. You still have the cost of the NICs and cabling but only need one port on the NIC.

Lastly, you could go all in with an enterprise switch like the HP ProCurve 5406zl which is a module hosting monster of a switch. These are massively upgradable but come with a lot of complexity to set up and manage. They are also incredibly loud (intended for server rooms) and suck a ton of power which generates lots of heat (thermal load). These are getting cheaper but are heavy to ship and still usually several hundred dollars with modules that can cost thousands. Don’t expect a warranty on these as they are being pushed out of enterprise usage as end of life.

1. Point to Point network

If you buy Dual port NICs then you can run a ring network between each node as direct connections. It significantly complicates your network config but may be worth it for the cost savings depending on your budget.

2. Switched network

Here are a pile of network switches that are both new and ancient that could support 10Gbps.

Price per SFP+ port is one metric.

Power consumption and acoustic (noise) are others to consider.

Mikrotik switches

Mikrotik CRS310-1G-5S-4S+IN 10Gigabit Switch 1G port 5 x SFP ports 4 x SFP+ port

MikroTik CRS309-1G-8S+IN Switch Gigabit Ethernet Port and 8x SFP 10 Gbps Ports

MikroTik CRS305-1G-4S+IN 5-port Network Switch 10Gbps Gigabit Dual System

MikroTik CRS317-1G-16S+RM Cloud Router Switch Rack-mountable Manageable Switch with Layer 3 Features

ZL Chassis

HP Procurve 5406zl Switch

ZL modules

J9309A - HP ProCurve 4-port 10GbE SFP+ ZL Module 2C3 (module)

This only gets you part of the way to the price.

Posts