INDEX
2024-01-25 - Storage and VMs (without virt-manager) So that plan with this one is figure out how to create VMs from basic config and test RAID Apparently you can export xml dumps from vms that can be used to rebuild system This uses "virsh" - controls any libvirt VM # https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-domain_commands-creating_a_virtual_machine_xml_dump_configuration_file#example-virsh-dumpxml # https://www.redhat.com/sysadmin/virsh-subcommands virsh start rh8-vm01 virsh shutdown rh8-vm01 virsh list --all virsh dumpxml rh8-vm01 # Also "edit" to edit xml config # https://linux.die.net/man/1/qemu-img qemu-img info OS-IMAGE-FILE # Gets overview of an image file https://drewdevault.com/2018/09/10/Getting-started-with-qemu.html https://wiki.gentoo.org/wiki/QEMU/Options curl -O ALPINE_STANDARD_X86_64_URL # Download iso image (can get alpine) pacman -S qemu qemu-img create -f qcow2 alpine.qcow2 16G # Create a virtual hard disk "alpine.qcow2" # Virtual disk is only as large as it needs to be - at base is about 200K qemu-system-x86_64 -enable-kvm -m 2048 -nic user,model=virtio \ -drive file=alpine.qcow2,media=disk,if=virtio \ -cdrom alpine-standard-3.19.0-x86_64.iso -sdl # Gives me "-sdl invalid option" pacman -S libsdl2 # https://wiki.archlinux.org/title/SDL # Still getting the same issues - maybe qemu-ui-sl package too Okay since this isn't working I'm going to follow the libvirt tutorial https://wiki.archlinux.org/title/Libvirt pacman -S libvirt qemu # install base libvirt (virsh) and hypervisor pacman -S dnsmasq # Add networking support usermod -a -G libvirt USER # Add my own user to "libvirt" user group (already exists) # "No need to create a new [polkit] group and rule file if your user is ... in wheel" systemctl start libvirtd; systemctl start virtlogd virsh -c qemu:///system # Test if working virsh pool-list --all # List storage volumes - /var/lib/libvirt/images qemu-system-x86_64 -vnc :N # Run on vnc port N (590N) e.g. -vnc :2 is 5902 qemu-system-x86_64 -enable-kvm -m 2048 -nic user,model=virtio \ -drive file=alpine.qcow2,media=disk,if=virtio \ -cdrom alpine-standard-3.19.0-x86_64.iso -vnc :1 # Now if you install remmina and libvncserver - open remmina and VNC:localhost:5901 # This gives you access to the server https://yuankun.me/posts/a-guide-on-running-alpine-linux-in-qemu/ # Run from CD (run from drive by removing the -cdrom bit) qemu-system-x86_64 -enable-kvm -m 2048 -nic user \ -drive file=alpine.qcow2,media=disk,if=virtio -cdrom alpine-standard-3.19.0-x86_64.iso -vnc :1 remmina -c vnc://localhost:5901 Since I've got that working I can get on the VM - default credentials are "root:" cat /sys/class/dmi/id/sys_vendor # Clearly running a VM in QEMU df -h # Looks like it's not actually storing any data persistently alpine-setup # Install alpine on drive - skipping network for now # https://wiki.alpinelinux.org/wiki/Alpine_setup_scripts#setup-alpine # https://www.qemu.org/2018/05/31/nic-parameter/ # -nic creates a frontend (-net) and backend (-netdev) # https://wiki.alpinelinux.org/wiki/Alpine_setup_scripts#setup-alpine # /etc/network/interfaces (open with vi) auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp hostname alpine-test rc-service networking --quiet start & # Don't need to configure dns due to dhcp settings setup-alpine # Configure networking before this to make sure it works # Install on vda with sys for simplicity (writes to virtual disk) ~200M Run qemu with "-monitor stdio" and you can enter system commands system_powerdown # graceful shutdown of OS # Shutdown system and reboot without CD iso apk add util-linux # Installs tools like lsblk lsblk # Looks like defaukts are vda1=boot vda2=swap vda3=data Now I want to make 3 new virtual storage drives and mount them qemu-img create -f qcow2 t1.qcow2 1G # And t2,t3 # In QEMU monitor screen: drive_add 0 if=none,id=t1,file=./t1.qcow2,media=disk,if=virtio # "Can't hot-add drive to type 7" - just going to mount at boot instead # ./run.sh qemu-system-x86_64 -enable-kvm -m 2048 -nic user \ -drive file=alpine.qcow2,media=disk,if=virtio \ -drive file=t1.qcow2,media=disk,if=virtio \ -drive file=t2.qcow2,media=disk,if=virtio \ -drive file=t3.qcow2,media=disk,if=virtio \ -vnc :1 -monitor stdio lsblk # Shows new devices are added Okay now drives are added I want to start making a raid array with them # https://wiki.alpinelinux.org/wiki/Setting_up_a_software_RAID_array modprobe raid1; echo raid1 >> /etc/modules-load.d/raid1.conf setup-keymap # This system apparently needs US keybind to use pipes fdisk /dev/vdb # n(...) t(fd) p w apk add sfdisk sfdisk -d /dev/vdb | sfdisk /dev/vdc # And same for vdd apk add mdadm mdadm --create --level=1 --raid-devices=3 /dev/md0 /dev/vdb1 /dev/vdc1 /dev/vdd1 cat /proc/mdstat # Show proress of raid array being made # Enable raid array at boot mdadm --detail --scan # Set up mdadm config from current setup rc-update add mdadm-raid; rc-update add mdadm boot; rc-update add mdadm-raid boot mdadm --assemble --scan # "Assemble array" whatever that means mdadm --detail /dev/md0 # Get info on array - need chunk size for formatting mdadm --detail /dev/md0 | grep "Chunk Size" # Returns nothing for me... # https://wiki.archlinux.org/title/RAID # stride = chunksize/blocksize = (default) 512KiB/4KiB = 128 # width = numdisks*stride = 3*128 = 384 mkfs.ext4 -v -L myarray -b 4096 -E stride=128,stripe-width=384 /dev/md0 mount /dev/md0 /mnt Okay going to watch that file system on my local machine and add some data What it looks like is t2 and t3 are at 1024M (set to max allocated capacity) But then t1 scales with the size of the contents - capacity is still 1024 Filled it to maximum with random data and it's really at 1000M locally So this is raid1 but I configured it for raid0 - need to look that up Apparently for raid1 this doesn't really matter Weirdly deleting all the data and then rebooting doesn't clear local space usage dd if=/dev/zero of=/dev/md0 # Zero out drive - does not affect qcow files # Scale down an image (repeat for all 3) item=t1; cp $item.qcow2 $item.qcow2_bkp && \ qemu-img convert -O qcow2 -c $item.qcow2_bkp $item.qcow2 && \ rm $item.qcow2_bkp
2024-01-27 12:00 - File Sharing Servers Going to make a basic NFS server in a couple docker containers - try out some features Just going to make a basic server image with nfs-utils and do the rest in bash replicas: 2 # Make 2 identical images I could try and mount a folder dependent on image but just going to mount read only for each Now I've got 2 replicas running - how can I interact with them separately? docker compose up -d docker compose ps docker exec -it nfs-server-1 /bin/sh https://wiki.archlinux.org/title/NFS # Config set in /etc/nfs.conf # Shares in /etc/exports /data/music *(rw,all_squash,anonuid=150,anongid=100) # Share folder as read write to anyone # Could do "SHARE IP(...) IP(...)" and set different rules for different server accesses exportfs -v # List current exports exportfs -arv # Push changes to running exports # Statr nfs-server.service In the end, there is an image for doing this but I prefer other means of testing Will try running in a VM
2024-02-01 14:20 - Fileshare in VM Reframing this project as running different file share protocols inside a VM So get a basic VM and run FTP/NFS/samba on it - configure it all using ansible First of all, how do I get a basic VM without having to do manual config at all Most basic solution is run the setup process once and keep a copy of the storage Or it looks like I can download cloud images (qcow2 files) for debian - try that https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-nocloud-amd64.qcow2 # Run the VM qemu-system-x86_64 -enable-kvm -m 2048 -nic user \ -drive file=debian-12-nocloud-amd64.qcow2,media=disk,if=virtio \ -vnc :1 -monitor stdio # Connect to it, as before remmina -c vnc://localhost:5901 Works perfectly - just log in with "root":"" and it works Next step is get ansible on this - run ansible server in docker and talk to client on VM So the control node needs ansible to be installed - but managed node does not Just had to reboot there for some reason - docker refusing to start (might have upgraded) # Use basic alpine image with ansible - from here can run arbitrary ansible commands docker run -it --rm alpinelinux/ansible:latest # compose.yaml name: filestore services: ansible: image: alpinelinux/ansible:latest volumes: - ./ansible:/ansible # Have an "ansible" folder for storing all config docker compose run --rm ansible sh # Open shell in container Now I have a docker image for ansible I need to link it to the VM https://wiki.archlinux.org/title/QEMU#Bridged_networking_using_qemu-bridge-helper # Should have qemu-bridge-helper installed by default with qemu echo "allow br0" > /etc/qemu/bridge.conf qemu-system-x86_64 -nic bridge,br=br0 ... # https://blog.christophersmart.com/2016/08/31/configuring-qemu-bridge-helper-after-access-denied-by-acl-file-error/ echo "allow all" | sudo tee /etc/qemu/${USER}.conf echo "include /etc/qemu/${USER}.conf" | sudo tee --append /etc/qemu/bridge.conf sudo chown root:${USER} /etc/qemu/${USER}.conf sudo chmod 640 /etc/qemu/${USER}.conf # This gets rid of "access denied by acl file" issue but not "No such device" # Need to actually create bridge device - https://bbs.archlinux.org/viewtopic.php?id=261207 # https://wiki.archlinux.org/title/Network_bridge#With_NetworkManager (as I use that) nmcli con add type bridge ifname br0 stp no nmcli con mod bridge-br0 ipv4.address 192.168.2.1/24 # Set it to any IP nmcli con mod bridge-br0 ipv4.method manual nmcli con up bridge-br0 nmcli con del # Run VM with "nic bridge,br=br0" and open console ip addr add dev ens3 192.168.2.2/24 # Set VMs IP to be 2.2 (host is 2.1) # At this point you can live add/rm IPs and test connections tcpdump -i ens3 # On VM monitor all traffic sent across the bridge ping 192.168.2.2 # From host machine, ping VM (should appear - also try 2.3) I've added all the host-side setup to the run script so this should setup as expected Now for the VM sid I'm looking at having some initial config to set static IPs https://www.debian.org/doc/manuals/debian-reference/ch05.en.html # /etc/network/interfaces auto ens3 iface ens3 inet static address 192.168.2.2 netmask 255.255.255.0 gateway 192.168.2.1 # Looks like it uses systemd-networkd - look at status - have to edit lib files instead # https://wiki.archlinux.org/title/Systemd-networkd networkctl # Show systemd-networkd managed networks networkctl reload # Apply changes to config files # /etc/systemd/network/09-static.network (was "10-") [Match] Name=ens3 [Network] Address=192.168.2.2/24 Gateway=192.168.2.1 systemctl restart systemd-networkd # For some reason systemd-networkd refuses to read any config files I set # So the reason is it defaults to using /run/systemd/network/10-netplan-all-en.network # Because it works in ASCII order "10-static" is overwritten by this - set to 09 # Now saving that file should allow me to communicate with it on boot - WORKS # Using run.sh I can now boot a VM and configure my system to talk to it in 1 command Following https://docs.ansible.com/ansible/latest/getting_started/get_started_inventory.html # ansible/inventory.ini [myhosts] 192.168.2.2 docker compose run --rm ansible sh cd /ansible ansible-inventory -i inventory.ini --list Okay now ansible can track the server it needs to be able to ssh By default debian cloud image seems to fail to run ssh sshd -t # Check ssh config of server - "no hostkeys available -- exiting" # https://unix.stackexchange.com/questions/642824/ssh-fails-to-start-due-to-missing-host-keys ssh-keygen -A; service ssh --full-restart # Now sshd is running we need keys shared between them # In ansible container, generate basic ssh key for root access to the VM ssh-keygen -f /root/.ssh/id_rsa # This is tedious to copy so might be best to just set up a user with password to copy it
2024-02-12 - Adding automated setup How can I make a small custom iso to store any scripts or info I want to transfer? # Install "cdrkit" for "genisoimage" # Create isodir/setup.sh which does all the VM side config genisoimage -o debian-setup.iso -R isodir/ # Mount this in startup script -cdrom debian-setup.iso Okay I've configured all that with some config variables in the run.sh script Only problem is the script needs to know where it's running to copy things # Some beautiful code copied from endPoint copied from somewhere else CURDIR=$(dirname -- "$(readlink -f -- "$BASH_SOURCE")") # Now run the script to start the VM and set up the server manually lsblk # Shows ROM is on sr0 mount /dev/sr0 /mnt That works - now need to get ssh keys auto generated for ansible Which means running the docker bit before the VM starts running # Keep a basic container running and kill it on demand command: tail -F /dev/null stop_signal: SIGTERM docker compose up -d docker compose kill # Stop it without care # In the VM setup script I need to make a user to link ansible # Luckily chatgpt can generate this quickly (runs as root so removed all "sudo") adduser ansible --disabled-password --gecos "" # Remove any prompts on generation usermod -aG sudo ansible echo 'ansible:changeme' | chpasswd mkdir -p /home/ansible/.ssh touch /home/ansible/.ssh/authorized_keys cat dockerpubkey > /home/ansible/.ssh/authorized_keys chown -R ansible:ansible /home/ansible/.ssh chmod 700 /home/ansible/.ssh chmod 600 /home/ansible/.ssh/authorized_keys # I can store the public key from docker in a file - data/dockerpubkey # Also want to exit if current user is not root (for protection) if [ "$(id -u)" -ne 0 ]; then echo "This script must be run as root" >&2; exit 1; fi Now I'm at a point where everything automatically sets up to a point where docker can ssh Just made the docker build make an ssh keypair and used "docker cp" to copy it to vm iso dir All that needs to be done is open the VM and run the basic setup.sh script # Ansible user doesn't need password to use sudo echo 'ansible ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/ansible I've also added some features to kill/wipe everything on closing, and run remmina at build Basically run the start script and everything builds from scratch But how do I add an nfs server using ansible? That was the initial goal Okay I want to share the folder "/srv/nfs/share" - will share ansible files when done Current issue is VM doesn't have access to internet - so no package downloads # Something in here fixed it (... = my network interface) sudo nano /etc/sysctl.conf # Add "net.ipv4.ip_forward=1" sudo sysctl -p iptables -t nat -L --line-numbers iptables -t nat -A POSTROUTING -o ... -j MASQUERADE iptables -A FORWARD -i ... -o virtbr01 -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A FORWARD -i virtbr01 -o ... -j ACCEPT iptables -L # Added "DNS=8.8.8.8" to network config file - internet now works # May have to add iptables config to main script too ansible-playbook -i /ansible/inventory.yaml /ansible/playbook_nfs.yaml # WORKS but missing ufw package - need to add to setup script too # And we can successfully mount: mount -v -t nfs 192.168.2.2:/srv/nfs/share /tmp/test
2024-02-17 - Testing features So I've got a working VM now - just need to try out different features on it I'm thinking autoSSH, dd, different file protocols, SELinux, LUKS - anything like that First trying LUKS - had this working on local machine but want to test deleting headers reset # Fixes shell when broken by invalid characters dd if=/dev/vda1 of=header.img bs=512 count=1 # Extract header of partition fdisk -x /dev/vda # shows much more info abou a drive # I want to LUKS encrypt an image (can be a file) and then extract the header # Install cryptsetup and lvm2 dd if=/dev/random of=test.iso bs=1M count=100 # Create 100MB iso file cryptsetup luksFormat ./test.iso # Encrypt it cryptsetup luksOpen ./test.iso base # Mount it pvcreate /dev/mapper/base; vgcreate base /dev/mapper/base # Turn drive into LVM volume lvcreate -l 100%FREE base -n home; mkfs.ext4 /dev/mapper/base-home # Format it mount /dev/mapper/base-home /mnt cryptsetup luksHeaderBackup ./test.iso --header-backup-file ./test.1.head # Backup header dd if=/dev/random of=/mnt/test bs=1M count=10 # Add random data to folder cryptsetup luksAddKey ./test.iso # Now backup is outdated cryptsetup luksClose base dmsetup reomve base # Remove device maps if refusing to close # Trying to unmount these things is a bit of a mess cryptsetup luksHeaderRestore ./test.iso --header-backup-file ./test.head cryptsetup erase ./test.iso # Wipes luks header content but not the whole header cryptsetup isLuks ./test.iso && echo true # Test - but better to just do luksDump wipefs -a ./test.iso # Seems to wipe header fully and yet works when using HeaderRestore Ideally we want to try some things in ansible too - try setting up selinux Added packages for SELinux, open port 5901, set enforce and reboot if state changed Also you can't combine different tasks naturally into 1 - oh well # Also adding UFW by default ansible-galaxy collection install community.general ansible-playbook -i ansible/inventory.yaml ansible/setup_selinux.yaml # Seems the VM has run out of space - how can I extend that? qemu-img resize debian.qcow 10G # Need to restart VM - this just extends the image size growpart /dev/vda 1 # Inside the VM, increase the partition size resize2fs /dev/vda1 # Then increase the file system size Currently the playbook is failing to open ports because "SELinux is disabled" May have to open ports after the reboot stage - not ideal as might shut me out Ah yes I've blocked out ansible with ufw - might have to manually set that ufw allow 22/tcp Haven't quite gotten selinux working using ansible but I'll figure that out next time Now I'm just going to go back and sort out the config so env data is shared between programs # Made a .env file containing all the key variables being used # compose.yml container_name: ${DOCKER_CONTAINER} # Don't actually need to reference .env - works already Now that's done, in theory I can change the image source and names of things arbitrarily # Made a little script to configure IPtables getting user input for the interface # Got the setup script using a generated set of environment variables (one source of truth) docker-compose down --volumes # Deletes only the rw part of the image (saves redownloading) # Also got the VM image only redownloading etc if it doesn't exist After some messing with it to make sure it works properly that project is done I have also written an ansible script to setup nfs, ftp and smb all in one - very basic So run that script and it configures all of them to point to "/srv/share" folder in the VM
2024-03-04 - Backing Just learned about a feature to layer qcow files like docker file systems (ro layer + w layer) Going to try and add that to this system - instead of having to backup the qcow image # Sample command (from chatgpt) qemu-img create -f qcow2 -b STATIC.qcow2 DYNAMIC.qcow2 # Extend a known image So all I need to do is instead of copying the image just run this command qemu-img create -f qcow -b ${VM_IMAGE_STATIC} ${VM_IMAGE} # qemu-img: debian.qcow2: Backing file specified without backing format qemu-img create -f qcow -o backing_file=debian.qcow2.0,backing_fmt=qcow2 debian.qcow2 # debian.qcow2 is now only 41K instead of another 348M # Going to try running the script without the generation # Seems much slower but it works - actually incredibly slow and failing to stop # Had to force kill it - might be a slight bug # Could not open './images/./images/debian.qcow2.0': No such file or directory # Had to stop using relative addresses and just cd to the directory - not an ideal workaround image_ro=`basename ${VM_IMAGE_STATIC}`; image_rw=`basename ${VM_IMAGE}` qemu-img create -f qcow -o backing_file="$image_ro",backing_fmt=qcow2 "$image_rw" qemu-img create -f qcow -b "$image_ro" -F qcow2 "$image_rw" # Slightly cleaner version In theory with this you could have 2 VMs running as replicas - with different rw layers So you could share the same base image Only thing that concerns me is this seems a bit unstable - that and the fs size is inaccurate killall -9 qemu-system-x86_64 # Force kill necessary for this # For now just going to keep it copying instead of using backing file https://fabianlee.org/2018/09/24/kvm-implementing-linked-clones-with-a-backing-file/
2024-03-04 - Fixing backing issues Now I'm essentially trying to fix this by looking into how qcow files work Cause currently you get kernel errors when you try to do too many writes Trying out guestfish https://man.archlinux.org/man/guestfish.1.en - libguestfs # Make backed storage device qemu-img create -f qcow -b debian.qcow2.0 -F qcow2 debian.qcow2 sudo guestfish --rw # Opens up a subshell for guestfish : add debian.qcow2 : run # Takes a while : list-filesystems : mount /dev/sda15 Okay from here in theory I can write a whole filesystem without needing any manual entry Going to try mounting it as normal instead - make "test" folder sudo guestmount --add debian.qcow2 --mount /dev/sda1 ./test # Read contents (but only with permissions) sudo ls test # Test IO usage - seems to be about the same - same issue with image file sort of catching up dd if=/dev/zero of=test bs=64k count=1k conv=fdatasync # Seeing some notes that this might fix it sudo sysctl -w vm.dirty_ratio=20 # Seems to stop it crashing sudo guestunmount test Now going to try with backing again May have just corrupted by own file system doing this - folder refuses to be deleted Reboot solved it - trying on unbacked file now to see the difference