Posts in category "hardware"

Upgrading to Windows 11 on my ThinkPad T480 laptops turned into a performance nightmare with some unexpected solutions.

My super lean Proxmox 8.3 testbed cluster running Ceph occasionally just decides to lockup a node based on it being incredibly limited on RAM and CPU. As much as I hate rebooting Linux/UNIX systems, this is a case where a nightly reboot of the nodes might help with reliability.

I want an extra ~350w of power available for a GPU that cannot run off PCIe bus power of 75w or 25w in some very old Dell Optiplex 990 Mini Tower nodes in my Proxmox cluster.

When one of my power supplies died earlier and I bought on eBay a NEW 750W Dell OptiPlex 9010 990 790 Power Supply Replace / Upgrade that was ~750w and the same form factor as those nodes PSU. This was just fast purchase to grab something that would ship the next day with no plan for an upgrade but I did pay attention that was both better and newer with a warranty.

So I have one machine that has the extra wattage available for a much better GPU like a Nvidia GeForce RTX 3060 12Gb.

I swapped my physical disks around in my low-end testing hardware cluster. I have a mixture of soldered to the motherboard eMMC and an external USB3 Thumbdrive serving for a root file systems and external /usr volumes now. I would like a quick performance check on reading and writing to those file systems. I also don’t want to setup a huge performance benchmark suite or additional tooling. I just want some quick results at this point.

My basic question is what did I loose in this decision to break out my /usr out to an external USB3 drive. How much performance did I loose?

My relatively new microwave just stopped heating things for no apparent reason one morning. We bought it a couple years back (about 3 years ago), so I was not happy as I expect these to work awhile with several working for ten (10) plus years. We picked up a new one from the local white box retailer as we wanted a replacement quickly. But my wife while digging around on Youtube found Microwave works but wont heat - Cheap and easy fix which was exactly what we experienced.

That video said it was likely a fuse or diode which are both cheap enough that they are worth an attempt at fixing. That will give me an extra microwave for the kids to use upstairs if I can fix it and save some landfill space.

Fuse	Diode

The Dell Wyse 3040 is a nifty little machine that is extremely small and a low power consumer. They are however not without issues. This is my foray into trying to get a couple of them working on my homelab. I bought a couple in early March 2024 based on watching Apalrd’s videos Dell Wyse 3040 Thin Client Teardown and Installing Proxmox VE 7 on Debian Bullseye with the various Dell Wyse thin clients.

I have collected nine (9) mostly functional Dell Wyse 3040 thin clients for use in my experimentation with Proxmox Clusters and SDN and Site-2-Site VPN configurations with Tailscale. Yes, I know I have a problem. :)

On the upside, they are very small low power consuming Debian 12 servers that have a 1Gbps NIC and run headless nicely once you fix the BIOS settings and Debian configuration correctly. What is not nice is their CMOS batteries are all mostly dying on me and their connector is a odd type that is not supported by many vendors and are between $8-$12 USD to replace. For example the Rome Tech CR2032 CMOS BIOS Battery for Dell Wyse 3040 is about $9.89 USD as of posting this. This bothers me intensely as the actual CR2032 can be picked up for well under a dollar ($1 USD) each at LiCB CR2032 3V Lithium Battery(10-pack) for a pack of 10 for $6 USD. Also, I’m picking these units up with power adapter for between $20 and $45 on eBay and the $10 bite jacks my price per unit up a good bit. So what to do?

We live in a world with a penny ($0.01 USD) per GB of storage. I just found this bare drive MDD (MD20TS25672NAS) 20TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5” Internal Hard Drive (for NAS, Network Storage) - 5 Years Warranty (Renewed) for $199.99 USD. I also found Avolusion PRO-X USB 3.0 External Hard Drive (Black) - 2 Year Warranty (20TB) for $229.99 $219.99 USD with the USB C enclosure.

Adding another SSD Drive

In my etermal tinkering with my Lenovo Thinkpad T480s, I have continued the trend of adding new features. So earlier, in A new to me but old laptop and New Laptop update, I threw out a bunch of enhancement options. Some of those I’ve done and some I left on the backlog as things that just cost too much on my metric of usefulness per dollar. The WWAN SSD for extra storage was one of those that just seemed like a bad bang-for-the-buck for storage. I also like the option to add a SIM card and have cellular network available in case I have to go back to consulting on the road.

Get access to switch console

You will need a console serial cable to get into your HP ProCurve 2800 switch.

Here is the one I bought from Amazon OIKWAN - USB Cisco Console Cable, USB to RJ45 Console Cable which has been useful on some other project as well. I have a break out for the RJ45 to let me use this on an old BlackArmor NAS and to interface with some robotics equipment. You mileage may vary but this one works great for me.

This is the ProCurve 2810-24 that is all 1Gbps ports with four SFP (not SFP+) ports that you can use with fiber or DACs. I bought three of these so I have a SAN and two home networks… then I picked up another one as a spare because it was less than $25 on eBay. So I’m all in on this switch for my home networks.

My new to me Thinkpad T480 is doing great. The better processor and the upgraded Nvidia MX150 GPU are both getting a workout with several of the new LLM models. I maxed out the RAM, Wifi, Hard Drive, and swapped around adding all the best components/features to one laptop consolidated from several different junkers I purchased. Those components include IR Camera, WWAN, backlit keyboard, good batteries, and a nice case.

I’ve been using an older Lenovo Thinkpad T460p laptop that I bought brand new May 2017 for $2210 USD with a pretty impressive discount for the time. The Thinkpad T460p released April 2016 over a year earlier from when I bought it. I over bought on the CPU/processor and bought extra power adapters and batteries upfront. It has been a great laptop but starting to show its age and I’ve been looking for a new laptop with a little more pep. Older USB is starting to be limiting and I’d like to go beyond 32Gb RAM.

This is a project I’ve been thinking about for a long time… how to get 10gbpe+ networking in a homelab without breaking the bank.

First option is just getting some DAC Cables and dual port 10Gbpe NICs then build a point-to-point ring network. That is relatively cheap and would set me up for future switched networks. DACs could be swapped out for GBics that use copper (thermal issues) or fiber (delicate).

Next is a relatively cheap at a couple hundred dollars switch with likely a low number of SFP+ ports. This is ~$200-$500 with anywhere from 2 to 16 ports at 10Gbps. Often switches with 10Gbps advertised only have one or two ports at that speed so shop carefully. You still have the cost of the NICs and cabling but only need one port on the NIC.

Lastly, you could go all in with an enterprise switch like the HP ProCurve 5406zl which is a module hosting monster of a switch. These are massively upgradable but come with a lot of complexity to set up and manage. They are also incredibly loud (intended for server rooms) and suck a ton of power which generates lots of heat (thermal load). These are getting cheaper but are heavy to ship and still usually several hundred dollars with modules that can cost thousands. Don’t expect a warranty on these as they are being pushed out of enterprise usage as end of life.

Hajo on the BlackArmor Forums has an older posting about getting Debian Linux 5.0 (Lenny) installed on BA NAS 110/220/4x0. This is not a port that includes the kernel but simply a minimum install that gets the system setup to install binaries out of the Lenny EABI ARM platform. The kernel that comes with the BA NAS is compatible with those binaries. The newer kernel for the Debian 6 or higher is not compatible with the BA NAS. This has some limitations but offers a way to get to some newer software pre-compiled. I don’t want to loose the existing functionality on my test system but the draw to DLNA services is pretty strong right now.

To top it off, Debian has a nicely setup cross-compilation setup documented for people working on non-Intel platforms. This offers a way to compile newer software without killing myself anymore on building the entire compiler and supporting software myself.

The goal has always been to make the NAS device useful and I want to play my movies off it to my TV upstairs so this might be the next thing I play with on the development NAS.

I was reading a bit more on DLNA servers and found that someone had gotten miniDLNA working under the Debian port for the BlackArmor NAS. NickolasZev looks like he is enjoying his Debian install on his BA110. He is documenting his Debian initial install, adding miniDLNA, a webserver, bittorrent and download manager. I’m not interested in putting Debian on mine and want to stay as close to the vanilla firmware as possible but it is nice to see that someone has it working on the hardware.

Today I was reading a couple of forums and bumped into a write up on the “BubbleUPnP - UPnP/DLNA Control Point and Renderer” application for Android that has both DLNA client and server support. The BubbleUPnP setup also has a Java based server mentioned in their section on NAS devices, such as the QNAP which is very similar to the BlackArmor NAS.

Earlier in my writing, I was reviewing a version of Java for the ARM that was called “Sun Java SE for Embedded”. It would allow for executing a Java application on the BlackArmor. Up until now, I did not have a Java application to really motivate me to install and test this but the BubbleUPnP DLNA server might allow for my BlueRay Play, Roku and Android cell phone to play my movies directly off the BlackArmor NAS.

There is one problem which is getting the ffmpeg library compiled and running. This is described in the docs above for the QNAP NAS but I’ve not been successful at getting a working toolchain yet. I may have to backtrack and see if I can just use the toolchain provided by Seagate to compile this newer version of ffmpeg.

Another issue is that the Java SE for Embedded is licensed strangely by Oracle. It cannot be redistributed and requires licensing costs if it is redistributed by a professional package.

So step one would be to compile the ffmpeg library and test it out. I hope to try that out with the existing Seagate toolchain. The next step would be to get the Java SE Embedded to run a basic HelloWorld application on the console. After that, try to get the BubbleUPnP server running.

I’ll see if I can get a couple hours together to work on this this coming week. It would be nicer to have an open source DLNA server but I’ll take what I can get. Since I’m talking about taking some time to compile C/C++ code anyway to get this working, I checked on open source DLNA servers and popped up something from eLinux that was interesting to read over. It looks like MediaTomb, uShare, and Serviio are candidates that could replace the BubbleUPnP server if I get the compiler toolchain working.

We will just have to see how much time I can get together to play around with this.

I’m clearing out about 90% of my old hardware from my home office.

Things like my ancient Sun Sparc Station 4 and 5 for my old Oracle work, and the older Sparc Classic (lunchbox), Sparc IPX systems used in my distributed web development days, and the Cobalt RaQ and iMac G3 systems I did development on about a decade ago. I’m also dumping a Compaq 286 laptop that served me well in doing assembly and C coding and it’s cousin the 486 that was my constant companion for several years. The dumping continues with numerous boxes of old cabling, strange and exotic old cards (ISA, PCI, MCA, EISA, VESA, Sbus, and some I don’t even recognize now), a couple of engineering sample video cards, some really old KVMs, couple of 24 port serial port concentrators (that I hacked the firmware on) and even a pair of old Cisco 2500’s used in my network testing days. It’s about 20 years of hardware exiting the house before we do the renovations and addition to my home.

It is really sad to see some of these little guys leave the house but looking forward to a cleaned out home office that I can get back to being a pleasant work and coding environment. So out the door they go.

In dumping all this equipment, I’ve been gathering the hardware junk box that I can re-use in other hardware projects. So far I have two of these boxes with weird stuff like an old Palm 5 docking station, some ear phones, some older coaxial cable, and the other random assortment of hardware junk you find useful when hacking hardware. These two boxes are starting too look really strange like something from a Mad Max movie.

This is cathartic writing about all these old pieces of equipment that were used in my professional and personal learning processes. It’s hard to part with some of these because of the happy memories associated with figuring out the next big problem in technology for that time.

Good bye little friends and now to start gathering the next set. :)

I found my old Samsung x426 USB cell phone programmer cable as I was cleaning out my home office. I might have mentioned this cable earlier when I wrote about adding a serial console to the BlackArmor NAS. The cable is a really old style one that has a weird connector for the cell phone way before the mini-USB became standard. The interesting thing about this cable is that it has a serial to USB converter chip (2303HXC 0546) that does the magic of converting serial to USB. That is why that thing was ridiculously expensive ($35) when I bought it back in the day.

Why this is even on my radar as possible is that the website CrapNAS had an entry for how to connect a serial or USB cable so you can watch the Linux boot up as a serial console session. They specify two different ways to do this. For the serial connection, they specify a MAX3232 as necessary. The USB connection has a schematic that includes a 2303HX. I’m not sure what the difference is between that and my 2303HXC so I will be doing some reading before I cut into the cable and get out the soldering iron.

Why am I even messing around with a serial console? Because as I am planning to mess around with the lower level system, I should have a back out plan if I do something wrong. A serial console on the device gives me more options during the boot up even if I cannot connect via a network connection.

As far as getting time to work on the compiler toolchain, I’ve been relaxing over the holiday break with family, cleaning my messy home office and have not even booted up the virtual machine since my last post. I intend to get back to it someday.

Yeah it’s been hectic at work and lots to do in domestic life as well so this hobby project hasn’t gotten much attention. Christmas break is coming however and I’m planning on getting some time to work on it. I think most Open Source projects or hobbies get a boost over the holidays.

Merry Xmas & Happy New Year in case I don’t get back here before the holidays.

Earlier I was working on the problem with compiling using the arm-elf-gcc on very basic programs. After some work, I found the base problem is that the libc replacement from newlib are not being picked up at compile time and properly using the syscalls mechanism. Hardwired hacks to work around it produced an executable but something is still wrong with how the compiler was built.

I’ve distilled the error down to a single search.

Google Search: "/lib/libc.a(lib_a-exit.o): In function `exit':" "newlib/libc/stdlib/exit.c:65: undefined reference to `_exit'"

Several other people appear to have run into the same problem, so I’ll be reading up on the problem and see what solutions exist.

The problem is that I can get an ARM executable but not sure why the default system is not working to produce a basic HelloWorld.

Two full toolchains built and a third that I still think might be made to work. The first is using older versions of everything and was mostly done as a test to get the build environment working against known sources that are known to build. Even this known build process required some effort to get working in a current OS environment. Those docs, notes and scripts will be coming in the near future.

I’ve been working on building a toolchain using the notes from OpenHardware Building the ARM GNU 4.3.2 with some success. I finally got the base set of GCC 4.3.2 tools to build successfully. I have not used the resulting GCC to produce an ARM executable or verified the executable works on the Black Armor NAS. Those are tests for tomorrow evening when I can get the NAS setup again on the network. It is currently in a box in the corner.

There were several minor things that needed to be updated and modified to get the scripts and environment to work. I’ve kept careful notes and will post those in the next couple of days once I’ve tested the output from the compiler works. I’m also attempting to update the versions of the libraries and software to more current versions as well. The GCC 4.3.2 and associated libraries are several years old and I’m trying to get the GCC 4.6.x to build along with newer newlib, binutils and insight/gdb using the same basic set of notes and scripts. I bumped into a zlib issues in the second phase GCC build that stumped me for the night. I’ll hit it again tomorrow. Again, I’m keeping careful notes and build docs for the newer version as well.

The dependencies from the operating system are sometimes a pain to track down for the software. I picked a very stripped down install of Ubuntu. The operating system I am using is Ubuntu Server 11.10 because it is easy to install and update. Any Linux would do but the package names may change. Ubuntu Server has no frills so you add everything you need which means all the libraries like GPM, etc.

So there is some progress and in the next couple of days I’ll let you know if the build produces working ARM executables. I’m really excited about getting a working “HelloWorld” out there.

Lesson learned on doing a toolchain build or anything else for that matter; make sure you are reading the most current documentation available. I was working with very old versions of the software by using the GNU ARM website mentioned earlier. In digging into the problems for those builds, I found a few other sites that have detailed discussions on building the toolchain for specific version of the ARM platform.

I found the GNU ARM Toolchain website awhile back and that they have several different versions out there for the toolchain. A toolchain is just the basic tools needed to build software. In this case it is the standard libraries, the compiler, debugger and various other tools needed to write software. The version of the toolchain that is provided by Seagate is version 3.0 and a very old version. The oldest on the ARM website is 3.3 with 4.1 being the newest.

Earlier I mentioned a technology called “Jazelle DBX” for the ARM processor that allows for Java Bytecode eXecution (JBX) directly in the ARM hardware which should make it run faster. That DBX technology is being phased out with newer Thumb-2 instruction set being the new preference by ARM for acceleration. However, the processor in the BlackArmor NAS was the first processor to have this Jazelle DBX feature and I want to see if it has any merit. I did some digging around, like I mentioned I would, and found that Sun had produced a version of Java that may have this technology.

So earlier I was digging around trying to find out more about the Black Armor NAS hardware and pulled some interesting information. Unfortunately, I don’t have a lot of ARM background so a good bit of it was confusing as I reviewed it.

Snippet from earlier hardware information gathering:

$ uname -a
Linux NAS3 2.6.22.18 #1 Thu Aug 26 12:26:10 CST 2010 v0.0.8 armv5tejl unknown
$ cat /proc/cpuinfo
Processor       : ARM926EJ-S rev 1 (v5l)

To rectify my lack of knowledge I started reading on Wikipedia and found the ARM architecture which made me realize that I’ve been missing out on an entirely different ecology of technologic innovation. The features that are available for each processor was an interesting ride down memory lane with my memory of Intel CPU features, that I’m familiar with, running parallel to the ARM decisions in the same areas. They have two completely different paths but seem to have interchange between the two. ARM has an interesting history as a company as well.

So my list of things to figure out keeps growing but I don’t seem to get any time to work on them.

1. UPS software setup
2. DLNA server functionality
3. USB Hub issue to figure out so I can run the UPS and Printer together.
4. Serial Port hack (new)

I also want to get the serial port hack working which requires some physical work disassembling the NAS and maybe some soldering work to build a serial converter. The major work is already done by another guy on http://crapnas.blogspot.com but I’d have to follow along. There also appears to be a shortcut with an old Nokia USB cell phone cable that might be worth checking out.

Maybe next weekend I’ll get some time. Project is just not getting cycles but I’m still thinking about it.

So I’m working on the UPS addition and forgot I use that USB port for my printer sharing. So time to do some reading on USB hubs and the Black Armor NAS 110. The printer is important and the UPS is important. I wonder if I can mix my peanut butter and chocolate.

Even stranger, there is a USB port on the front of the box but that is special purposed for just USB Memory backups only.

So I have several paths here to check.

Here is the beginning of a dump of information on the Black Armor device from the Linux kernel and environment. From this I learned the processor type and features. I also got some pointers to cross-compiler options used. These will all be important later.

I figured out something simple but neat on the Black Armor NAS 110 (BA-NAS110) device. It has rsync a powerful file-system replication tool from UNIX.

Caveats are that in order to do this you must have root on the device and a ssh connection with the command line. I’ll write a friendly doc on how to get ‘root’ later. (Just search for Hajo Noerenberg’s work on the subject sans the friendly write up if you want to do it now.)

So, the BA-NAS110 is capable of using rsync from the command line to replicate its data to another NAS or Linux system if you have root on the system. Getting it setup was simple enough but knowing that the rsync daemon and client were on the systems was the trick.

You have to create a rsyncd.conf file since there isn’t one pre-built. Syntax is common to the typical rsync 3.0.4 version.

I found something fun.

The Seagate NAS (Network Attached Storage) that I’ve been using at my house is running an embedded Linux. A NAS is a big network hard drive you can share between computers. I got a root account on it and have found a whole world of fun that could be done in there. Root is the master administrative account for UNIX systems that let you do extra things beyond the normal.

First steps is getting a functional toolchain and then build some trivial tools. The goal would be to have a full set of GNU tools available in a package format for people to use. I want to publish a full working OpenSSH with scp support and rsync for this thing as a starting point. Maybe add some features for NFS. Just digging around on this thing reminded me how much I enjoy hacking on hardware.

I bought new cell phones (Motorola Razor v3) for the whole family and I’ve just gotten them connected to my laptop. I’m trying to get my calendar and contact list on Gmail to download to it. The MPT (Motorola Phone Tools) lets you tie Outlook or Lotus Notes to your phone. The 5th gen video ipod has a tool to convert Gmail Calendar into your iPod. That might be an interesting place to start. It would be nice to tie the calendar together for the whole family on our phones.

The phone itself is actually quite a powerful little toy. The mobile java is an interesting environment.