Next bit of the Home Lab update – a 10Gbs link between the 2 servers. For less than £50?
Mellanox MT26448 ConnectX 10GbE. (PCIe 2.0)
From the US. £48 delivered. In 8 days. 2 cards, 2 cables with SFP+ on each end. Courtesy of eBay.
installation went like this:
- Shut down server and unplug
- remove top cover and remove daughter-board
- Install card on low-profile side
- re-install daughter-board
- close server and reconnect power
- Fire It Up!
ESXi 6 detected it and installed the drivers without any intervention, happily displays 10GbE.
Same for the other server. Hook up the cable and lo and behold – 10Gbs goodness in your system.
First test – vMotion. 2 new vmk ports – 192.168.120.1 and 192.168.120.2 with a mask of 255.255.255.252. Swap the vMotion traffic over. I had used vMotion over the USB ports and it was pretty quick taking 30 – 40 seconds for a hefty vm. Over the new 10Gbs it went….. 2%… done. In about 5 seconds!
But its not all speed I’m interested in, its being able to run a number of VLANs and networks across it. This is a lab after all. I need a couple of decent SSDs and more RAM and VSAN is next on my target list.
To go along with VCSA 6.5 HA. I set that up this afternoon – running over 10Gb. Works really well.
The long-term plan is to find a managed switch with 4 x SFP+ and 24 or 48 copper ports then I can expand what I can do with 10gbs. With another 2 cards and cables……
Carrying on from my previous post – Home Lab – reloaded! I thought I would detail the additions I’ve made.
The first was a follow-on from a post on VirtuallyGhetto – William Lam’s excellent blog.
He has a post detailing how to set up a USB 3.0 single or dual NIC adapter.
I have followed the instructions and can confirm that it works a treat and takes 2 minutes to install. Download the VIBs from Williams page, follow the instructions and Robert is your mother’s brother!
I used the Dual-port StarTech – £50 from Amazon. USB 3.0 has a throughput of 5Gbs. So 2 x 1Gbs NICs are not exactly going to push it are they? Network speed is excellent, although I haven’t monitored it yet so don’t have the numbers. But I do now have 4 x 1Gbs ports.
The StarTech adapters also support LACP, Jumbo frames (4000kb max), and lots of other goodies. So if you are NIC-challenged – its an easy setup and an easy win.
Right. I have a new home lab. Still have the old one but the new one is SO much better.
I decided to bite the bullet and buy new. Restrictions included:
- Price – it is a home lab after all
- Power consumption
- Normal operating volume
I really liked the Supermicros but at the price point and with DDR4 being the price it is, they were just too expensive. So before Christmas I did some extensive research, and stumbled upon the HPE DL20 Gen 9. They are on the vSphere 6 HCL, They use the E3 v5 Xeons, and its the cheapest server I could find with the E3-1230-V5, the lowest powered of the E3 family with hyperthreading. So I bought 2. And was surprised to find them arrive with rapid-rail mounting kits! Its all good! And I am a very very happy camper!
And as far as the WAF goes – well she said that I could dip into savings to build my lab, but it MUST be future-proofed. And I MUST be happy with it afterwards. Would be rude not to comply. After all….
The story so far:
- 1U shallow 15″ depth.
- 2 x PCIe 8 slots on daughter board
- 2 x M2 headers on the Mobo
- 2 x 1gbs NICs – one shared with iLO
- Mine came with 16GB – 1 x DDR4
- They are whisper-quiet, the fans do spin up occasionally and it sounds like they are breathing. VERY Quietly!
- Booting off a SanDisk 32GB USB 3.0 drive from the onboard socket
- ESXi 6.0U2 – HP Custom Image
- No CD but who uses them these days? I have a USB BluRay player anyway if I need it.
Only negative is the inclusion of 2 x USB 2 ports on the front – but no VGA. Odd. 2 x USB 3.0 on the rear with a VGA there though. In a tight rack its a tad frustrating having to pull the server out to struggle to plug in the monitor though a small gap at the back in but I have resurrected an old 2-port KVM to use with them so that’s not necessary any more.
They come in 2 storage configurations – 2 x 3.5″ SAS and 4 x 2.5″ Hot-swap SAS. I opted for the 2.5″ hot-swap for greater flexibility. However no drive caddies. This IS HP don’t forget. However – Amazon to the rescue – £12. Not HP genuine but have the spinney round LEDs. I bought a pair of WD Red 750GB 2.5″ NAS drives. There are reasons for this:
- They are SATA – but consumer-grade SATAs are not designed for 24×7 operation. The WD NAS Drives are.
- They are designed for hot-swap and RAID – so can handle failing properly and being removed from a RAID without taking the whole set down.
- I already have 2 of the 3.5″ WD Reds in 3TB versions and really like them.
- Its a HOME LAB not an enterprise production datacenter. SAS drives are just too expensive. These were a good price. Don’t care about massive performance differences.
- NAS can present NFS datastores and run VMs happily. So these should do fine.
- I also run iSCSI from both servers to a 3TB WD Red in a Synology NAS.
This is the basic setup. I will add other posts to separately detail the further enhancements – USB 3.0 NICs, 10Gbs NICs, VCSA 6.5 and so on.
These little servers just keep giving and I’m really happy with them.
Windows 2012 Server has been around for a while but like many IT professionals I haven’t had much exposure to it. I’ve spent most of my time recently on 2008 and upgrades from 2003. I don’t actually need 2012 Server myself, so how do I get experience with it? Enter AWS! I’ve signed up for the 12-month free trial and am running an instance of 2012 Server R2 to get a feel for the new features, and look and feel of it, but also a good opportunity to have a look at AWS. Seems ok so far, and the vm runs fast enough for my needs. RDP session is stable and quick even though I’m in the UK and it – according to Amazon – is in Oregan, USA. Why? No idea – I’d have thought Ireland was a more efficient location. But it’s their system….
Windows 2012 Server R2 seems ok, but still not a fan of the flat tile interface and that you still seem to have to click in a lot of places to get the simplest of things done but there seem to be some good improvements under the hood.
Well – vSphere 6.0 is upon us. Almost. I watched the EMEA live launch last Tuesday, and it’s pretty good from what I saw. There are some really good new features and the maximums have been lifted yet again. No documentation out yet though.
The massive improvement in the vCenter appliance capacity – 1000 hosts, 10,000 vms. I’ll be using that now rather than the Windows-based install. No Windows license or CALs to worry about. Gotta be good!
SMP for Fault Tolerance makes it actually usable. A single CPU is a bit pointless but 4 is much more realistic.
High-latency or long-distance vMotion. At Last! Being able to vMotion between vDCs and vCenters; across large geographical distances.
I really like the Fault Domains – Rack Awareness as they called it. I could have used that and more of these new features in a project I was involved with a couple of years ago.
So much really good stuff to play with there. I believe it’s available from March so will be getting hold of a copy for my lab.
For a long time I’ve wanted to build a home VMware lab for working toward the VCP qualification. Having taken a step back this year from other commitments that have taken up huge amounts of time for the last 3 or 4 years I now want to aim to get it out of the way by the end of Q2 this year.
I was going to use the Dell 1U Cloud Server that I bought a couple of years ago, but it’s simply too loud and wifey won’t let me run it for any decent length of time- and besides you don’t need the heating on when it’s running! I was also going to use my Synology NAS as I have an old 100GB iSCSI target on it already.
So last year – as I do a lot of this stuff for my job – I bought a Lenovo W540 laptop with a whopping 32GB RAM and a 4-core i7 (8 cores with hyper-threading) specifically for running vms. I work a lot with VMware so need something to run vms on. It also has a CD replacement 1TB hard drive. I bought a copy of VMware Workstation 10 and had a play. The 1TB drive frankly is way too slow. So after a bit of research into the speed differences between Thunderbolt and USB 3.0 – I bought a 500GB Crucial M4 SSD and a USB 3.0 caddy – there’s very little between them. Having loaded up the drive, I can run a full AutoLab off the SSD with practically no loss of performance.
But then it went missing.
So I had to start again. I bought another 500GB M4 and another USB 3.0 caddy, Only this time I’m going to do things a bit differently.
Having signed up for the new VMUG Advantage program, I got myself a 365-day licenced vCenter 5.5. I already have the relevant vSphere client, but though I don’t like it I really aught to get to grips with the new web client. I have 2 ESXi 5.1 hosts – I want to go a proper upgrade path to 5.5 U2 and have VUM running properly.
I had a look at the way a lot of people ran their labs and finally settled on something akin to this one:
My current setup is as follows:
- VMware Workstation 11 (Really don’t like the new interface – looks like an old CDE or Win 3.1 icon set.)
- DC – Windows 2008 R1
- Domain: Lab.local – I know – so unimaginative. But short to type!
- OpenFiler NAS with 40GB iSCSI and 10GB NFS shares
- 2 nested ESXi 5.1 hosts – build 799733 – lots of updating to do for practice.
- 4GB RAM
- 2 vCPU – 2 cores each. Because I can…
- iSCSI and NFS datastores
- vCenter 5.5
So that’s it for now. I have some small Linux OVFs to try out on the hosts.
That’s it for now. I’ll post updates as it goes along.
Well – many thanks to my kid brother Nick, a graphic designer. He’s been working on the logo and branding for me and done a great job. Very happy with the results: versatile logo, smart-looking stationery and the banner on this site.
You can catch up with him at www.hallidaybooks.com
Whilst preparing some of the pages for this site I began thinking about my journey into IT and how my outlook on technology has changed.
My computing history ran alongside an engineering career, something I still have a deep love and affection for. I am still fiercely proud of my engineering skills and experience.
In the 1980’s I bought a new Commodore Vic20 which I had for several years before acquiring an Apple ][e with 2 x 5.25″ external floppy drives that the monitor sat on. That gave way finally to the onslaught of IBM and an XT (that I still have) took pride of place. A whopping 2MB RAM – power computing! Then began the x86 climb along Moore’s law. I progressed to a 268, 386 – and that was joined on the motherboard by a 387 co-processor. I thought it was cool, anyway!
486 SX -> DX (different versions) gave way to Pentiums. Brings tears to my eyes just remembering! Especially installing Windows 3.11 from 500 (ok, 9 – I think, 7 with the OS and 2 with the network drivers if memory serves) floppy disks while my wife was shouting at me to leave it alone and come and have dinner! Ahh – happy days!
Well, that’s changed again and again over the years. On the early 486 I ran SuSE 5 – again I still have the 5 CDs that it came with. Early KDE – that was soooo slow!