KVM Lab Setup
Main components:
- kvm
- qemu
- libvirt
Client tools:
- virsh
- virt-manager ( Virtual Machine Manager )
- virt-install
- virt-viewer ( uses VMC and SPICE )
Other tools:
- dnsmasq
- dhclient
- ebtables
- bridge-utils
Qemu Stuff ( extra / optional )
Install qemu packages.
sudo apt install qemu qemu-utils qemu-system-x86
Creating images and VMs with qemu:
qemu-img create host1.img 20G
qemu-img create -f qcow2 host1.img 10G
qemu-system-x86_64 -hda ubuntu.img -boot d -cdrom /home/user1/Downloads/rhel-8.4-x86_64-dvd.iso -m 1000
qemu-system-x86_64 -hda ubuntu.img -boot d -cdrom /home/user1/Downloads/rhel-8.4-x86_64-dvd.iso -m 1000
KVM - CPU / Hardware support
Check for CPU virtualization support ( Intel VT-x or AMD-V ):
egrep -c '(vmx|svm)' /proc/cpuinfo
0 # cpu doesn't support hardware virtualization
1 or more # cpu does support hardware virtualization ( still needs to be enabled in bios )
Also check if it is enabled with this:
sudo apt update
sudo apt install cpu-checker # probably already installed
kvm-ok
KVM and libvirt Setup
Install important virtualization packages:
sudo apt -y install bridge-utils cpu-checker libvirt-clients \
libvirt-daemon virt-manager virtinst qemu qemu-kvm
Make sure libvirtd is active:
sudo systemctl start libvirtd
sudo systemctl enable libvirtd
sudo systemctl is-active libvirtd
Give my non-root user permissions to work with virtualization tools:
sudo usermod -aG libvirt $USER
sudo usermod -aG kvm $USER
| /var/lib/libvirt/ | libvirtd files kept here |
| /var/lib/libvirt/images | images kept here |
| /var/lib/libvirt/isos | can create this for ISOs (non-standard but good idea) |
Creating and Cloning VMs
Pull down an OS install ISO:
sudo wget -P /var/lib/libvirt/isos \
https://releases.ubuntu.com/22.04.3/ubuntu-22.04.3-live-server-amd64.iso
VMs can be created using the virt-manager GUI or the virt-install CLI tool.
Create a VM using virt-install:
sudo virt-install \
--name host1 \
--ram=2048 \
--disk size=10 \
--vcpus 1 \
--os-variant ubuntu22.04 \
--cdrom /var/lib/libvirt/isos/ubuntu-22.04.3-live-server-amd64.iso
Pure text based console installation ( no graphical VNC or SPICE ).
Remote installation media can be specified with ‘–location’ and will need to be used because ‘cdrom’ is not compatible with –extra-args=’console=ttyS0’
sudo virt-install \
--name host2 \
--ram=2048 \
--disk size=10 \
--vcpus 1 \
--os-type linux --os-variant ubuntu22.04 \
--location 'http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/' \
--nographics \
--extra-args='console=ttyS0'
virsh list --all
virsh domifaddr host1
ssh 192.168.122.166
Configure guest system to work with console after installation:
sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash console=ttyS0"
sudo update-grub
sudo reboot
virsh console host1
ctrl - ] # to exit
More options:
....
--disk path=/var/lib/libvirt/images/u19.qcow2,size=8 \
--location 'http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/'
--extra-args "console=tty0 console=ttyS0,115200n8"
--extra-args='console=ttyS0,115200n8 serial'
--console pty,target_type=serial \
--network bridge=virbr0,model=virtio
--graphics none
--os-variant generic
...
Clone a VM:
virt-clone \
--original host1 \
--name host2 \
--file /var/lib/libvirt/images/host2.qcow2
List all:
virsh list --all
Serial Console:
virsh ttyconsole my_vm
virsh domifaddr vm1 # get ip address
ssh-copy-id rkamradt@192.168.122.95 # use ip address from previous step
virsh domifaddr node1
Libvirt URIs ( which hypervisor to connect to ):
| qemu:///system | system libvirtd instance ( default for virt-manager, Openstack, oVirt ) |
| qemu:///session | session libvirtd instance ( default for virsh, gnome-boxes, libguestfs ) |
When not logged in as root virsh may default to using qemu:///session which may be different from virt-manager.
Specify the system URI:
virsh --connect qemu:///system create host1.xml
Create a user specific config which can be used to point virsh to qemu:///system:
sudo cp -rv /etc/libvirt/libvirt.conf ~/.config/libvirt/ &&\
sudo chown ${USER}:${USER} ~/.config/libvirt/libvirt.conf
Commands / Operations
Some commands I used:
vi alice.xml
virsh create alice.xml
virsh start alice
virsh list
sudo brctl addbr br0
sudo brctl show
More commands I used:
sudo netstat -nap | egrep '(kvm|qemu)' # find vnc port
virsh destroy alice
virsh shutdown alice
virsh suspend alice
virsh resume alice
virsh console alice
virsh autostart alice
virsh dominfo alice
virsh edit alice
virsh undefine alice # actually delete vm ( shutdown before or after )
Couple more commands:
virsh domrename vm1 template # rename a VM
virsh # basically run a virsh shell
VM Management Tips
virt-clone --original vm1 --name vm2 --auto-clone
virt-clone --original vm1 --name vm3 --auto-clone
Create 10 clones:
for i in 0 1 2 3 4 5 6 7 8 9; do virt-clone --original template --name vm$i --auto-clone; done
Get MAC address:
virsh domiflist vm1
virsh domiflist vm5|grep -i bridge| awk '{ print $5 }'
for i in `virsh list --all| tail -n +3| head -n -1 | awk '{print $2}'`; do virsh domiflist $i|grep -i bridge| awk '{ print $5 }'; done ## list all MACs
Start all VMs:
for i in `virsh list --all| tail -n +3| head -n -1 | awk '{print $2}'`; do virsh start $i; done
Stop all VMs:
for i in `virsh list --all| tail -n +3| head -n -1 | awk '{print $2}'`; do virsh stop $i; done
Generate DHCP Configs:
for i in `virsh list --all| tail -n +3| head -n -1 | awk '{print $2}'`; do virsh domiflist $i|grep -i bridge| awk '{ print " host test1 {" $5 }'; done ## list all MACs
host test1 {
hardware ethernet 08:00:27:85:AE:C9;
fixed-address 192.168.3.150;
}
Map IPs to MACs in DHCP:
Almost except that 10 is wrong, need better counting. Use IP pools in Python: ( also maybe exclude the template and create a pool for names ):
n=0;echo $n;for i in `virsh list --all| tail -n +3| head -n -1 | awk '{print $2}'`; do echo "host test$n { hardware ethernet " `virsh domiflist $i | grep -i bridge | awk '{ print $5}'` "; fixed-address 192.168.3.15$n; }"; let n=n+1; done | sed 's/ ;/;/'
Don’t really need this once it is added to the template:
cat ~/.ssh/id_rsa.pub | ssh user1@hostname 'cat >> .ssh/authorized_keys'
OR
ssh-copy-id -i ~/.ssh/id_rsa.pub user1@host
for i in 51 52 53 54 55 56 57 58 59 60; do ssh-copy-id -i ~/.ssh/id_rsa.pub user1@192.168.3.1$i; done
Bridged Network Setup
Show network bridge info:
brctl show
Networking:
sudo brctl addbr br0
Make sure your guest VM requests a DHCP lease by MAC and not hostid
/etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp1s0:
dhcp4: true
dhcp-identifier: mac
version: 2
Disable netfilter on bridges
/etc/sysctl.d/bridge.conf
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-arptables=0
/etc/udev/rules.d/99-bridge.rules
ACTION=="add", SUBSYSTEM=="module", KERNEL=="br_netfilter", \
RUN+="/sbin/sysctl -p /etc/sysctl.d/bridge.conf"
Remove default kvm interfaces:
virsh net-destroy default
virsh net-undefine default
Remove with these commands if the interfaces haven’t been removed:
ip link delete virbr0 type brigde
ip link delete virbr0-nic
Static config from enp0s7 is moved to br0:
/etc/netplan/00-installer-config.yaml
network:
ethernets:
eno2:
dhcp4: false
dhcp6: false
# add configuration for bridge interface
bridges:
br0:
interfaces: [ eno2 ]
addresses: [192.168.3.22/24]
gateway4: 192.168.3.1
mtu: 1500
nameservers:
addresses: [8.8.8.8,8.8.4.4]
parameters:
stp: true
forward-delay: 4
dhcp4: no
dhcp6: no
version: 2
sudo netplan apply # apply
ip a show # check
host-bridge.xml
<network>
<name>host-bridge</name>
<forward mode="bridge"/>
<bridge name="br0"/>
</network>
virsh net-define host-bridge.xml
virsh net-start host-bridge
virsh net-autostart host-bridge
virsh net-list --all
Change interface for dhcp server:
sudo vi /etc/default/isc-dhcp-server
INTERFACESv4="br0"
Add VNC to Existing KVM Guest
Edit the VM config:
virsh edit host3
Within the
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
- stop / start the VM if needed
Terraform
KVM buildout with terraform
Troubleshooting
If you have any issues:
- if any error
- log out and back in ( from terminal or also from desktop especially if launching virt-manager from a menu )
- check if you can write to this:
sudo ls -la /var/run/libvirt/libvirt-sock
If you have problems creating a VM:
sudo chown root:libvirtd /dev/kvm
Relogin or restart kernel modules
lsmod|grep -i kvm
rmmod kvm
modprobe -a kvm
Might be needed if you have another hypervisor like VirtualBox
virsh --connect qemu:///system create host1.xml