j a m i e

The easy way to update Dell servers.

If you’re reading this post you’ve likely been through the same battles I have;

Use the dell lifecycle controller to just update via FTP.
Use a flash drive inside the lifecycle controller or network shares.
Use Dell repo manager to do it yourself.

Bollocks to all that, none of it ever works properly. The FTP never works at all and anything else is just making stuff too complicated so here’s what you do.

Go here, in fairness to Dell they do give multiple workarounds but it would be nice to just have one port of call. Click on the model you need to update and download the newest update ISO they have – they should all have isos located at dell.box.com.

Download Rufus, and burn the ISO to a new flash drive.

Take down your hypervisor, or whatever you’re running and reboot, hit f11 to check boot options then boot from the new flash drive.

All going well, this will work just fine for you. However, usually this doesn’t go just fine and indeed has never worked for me, apply_bundles.sh is invalid.

If you get that error message hit alt +f2 to open a new shell and lets get started.

lsblk to show all drives, find the one that resembles your flash drive – check the storage sizes. It will likely be sdb1, sdc1 etc

mount /dev/yourdevice /opt/dell/toolkit/systems
cd /opt/dell/toolkit/systems/drm_files
./apply_bundles.sh

You might have to reboot and run it 2-3 times, usually it updates everything for me first time. If the screen goes black - don't do anything - just wait, it just takes a while.

Enjoy.

Loud Dell R410 fans and how to fix them.

Ever find one of those issues where you’ve read what feels like, 100+ articles, threads and walkthroughs on how to keep the noise down on the jet-turbines that live inside your R410 – Only to find out nothing works?

Then do I have a solution for you!
What we’re going to do is setup IPMI and run a quick script over cron to keep your server quiet.

Login to your DRAC and go to the IPMI settings (bottom right of screenshot)

Scroll down, enable it and hit apply.

Next up we need something to call it from, I used my zabbix/ansible server as it’s always online anyway.
Install ipmitool on it, apt install ipmitool
Create a new folder for your scripts, I just used a folder within root.
mkdir /root/ipmi/ and create your script nano whatever.sh

#!/usr/bin/env bash
# IPMI SETTINGS:
# Modify to suit your needs.
IPMIHOST=1.1.1.1
IPMIUSER=root
IPMIPW=yourpassword

echo "Setting fan speeds to 4k RPM"
ipmitool -I lanplus -H $IPMIHOST -U $IPMIUSER -P $IPMIPW raw 0x30 0x30 0x01 0x00
ipmitool -I lanplus -H $IPMIHOST -U $IPMIUSER -P $IPMIPW raw 0x30 0x30 0x02 0xff 0x1a

Edit the IP along with the DRAC password. The first line beginning with ipmitool enables manual fan control, the second line sets it to its new speed. This works for 4k rpm for both R710s and R410s which is all I've tested on. Please do your research on the correct hex that you want to use beforehand!

Some examples are;

3000 RPM: raw 0x30 0x30 0x02 0xff 0x10
2160 RPM: raw 0x30 0x30 0x02 0xff 0x0a
2160 RPM: raw 0x30 0x30 0x02 0xff 0x09

Afterwards, monitor it for a bit and if you're happy with it, set it up in a crontab to fire every 5 minutes to keep it in line.

*/5 * * * * /root/ipmi/whatever.sh >/dev/null 2>&1

Enjoy your new quieter homelab!
As a sidenote, my Dells were at 12-14k RPM because of expansion cards being used, this brought it from a high pitched whine to a nice hum!

Hooking zabbix api into grafana.

Eventually, I’m going to setup influxdb and telegraf for everything to get more accurate (and faster reporting) but for right now, here’s a good way to get grafana setup with some graphs. That’s all we care about anyway.

Create your new VM or login to whatever will be hosting your grafana instance.

nano /etc/apt/sources.list and add this

deb https://packagecloud.io/grafana/stable/debian/ stretch main

Save & close. Next, add the key;
curl https://packagecloud.io/gpg.key | sudo apt-key add -

And install grafana sudo apt update && sudo apt install grafana

To get it to start at boot
sudo systemctl daemon-reload
sudo systemctl enable grafana-server.service

And start, sudo systemctl start grafana-server

You can access it afterwards at http://whatever-ip:3000, works out of the box with reverse proxies too, super straightforward.

To install the zabbix plugin,
grafana-cli plugins install alexanderzobnin-zabbix-app and restart grafana afterwards service grafana-server restart

Login to grafana (admin/admin) and go to plugins, then Zabbix and select Enable.

Next, we have to configure it to poll the zabbix api.

Up to date versions of zabbix have the API enabled for all users so it should work out of the box. Don’t forget to add /api_jsonrpc.php to your zabbix url.
You can stop here if you wish, but leave ‘trends’ and ‘direct-db connection’ disabled.

If you want to set these up you’ll need to add mysql as a remote data source. Luckily, that’s pretty painless too. Click on Settings, Data Sources, Add data source and select mysql.
Next you’ll need to login to your mysql server and create a new user and grant it access to the zabbix database;

mysql -u root -p
create user 'grafana-zabbix' identified by 'yourpassword';
grant select on zabbix.* to 'grafana-zabbix';
flush privileges;
exit;

Then hop back to the mysql data source page, hook it all up.

Afterwards, you can go back into the zabbix plugin and enable 'trends' and 'direct db connection' and you're all set.
This plugin has several templates you can import to speed up the dashboard-building part, I'd suggest importing them to get a grip on what you want, then take it from there.

You can check out my public instance here.

Increasing the size of a ubuntu VM in ESXi

I’ve been putting this off forever because I’ve been lazy but got some time today so I finally got around to it. As it turns out, it’s not as awkward as I thought it was going to be!

So first things first, you want to increase the disk space allocated by ESXi, select your VM, click edit and set it to whatever you need.

Reboot the VM and type cfdisk.

Select the Free Space, hit enter and hit enter again on the partition size.

Next, navigate to write, hit enter, type yes.

Next we need to change type, select type;

And write the changes again. Then quit and reboot the VM.

After restarting, create the new volume pvcreate /dev/sda3

Physical volume "/dev/sda3" successfully created

Type vgdisplay to get the volume we'll be extending.

vgdisplay
  --- Volume group ---
  VG Name               database-vg

And to extend, vgextend database-vg /dev/sda3

  Volume group "database-vg" successfully extended

Extend root, lvextend -l+100%FREE /dev/database-vg/root

Size of logical volume database-vg/root changed from 3.52 GiB (901 extents) to 8.52 GiB (2181 extents).
  Logical volume root successfully resized.

df -h to get the filesystem

/dev/mapper/database--vg-root  3.5G  2.4G  856M  74% /

And extend this too, resize2fs /dev/mapper/database--vg-root

Lastly, df -h again to see your extended vm.