assumptions made in this document: the server has one network card, otherwise create the VM with the same number of NICs in order to populate all of them with correct prefixes (combined ip address and netmask; 12.34.56.78/24) the server has one hard disk, otherwise decide what to do with the remaining disks (use as separate encrypted storage volumes or raid) whenever $VARIABLE appears in commands below, it should be substituted with the correct value lines starting with # are commands to execute as root start by booting the server in rescue mode, preferably something debian-based to avoid getting warnings from ssh about malicious keys, use this to ssh from your workstation into the rescue env # ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username@ipaddress ON THE SERVER (RESCUE MODE) ########################### assuming you are using online.net as provider, here's a section on slightly improving the terrible rescue env (blank line inbetween steps that require a manual action to continue) change the language to english, disable the 400 whitelisted ssh keys, and then change the password for the regular user # export LANG=en_US.UTF-8 # echo 'export LANG=en_US.UTF-8' >> ~/.bashrc # mv ~/.ssh/authorized_keys ~/.ssh/nope # passwd make yourself root # sudo su then disallow ssh keys here too and change root password # mv ~/.ssh/authorized_keys ~/.ssh/nope # passwd list all services which are listening for remote connections # netstat -tulpn | awk 'BEGIN {cmd="kill -9";meta=" "} on {chrome=substr($0,1,nch-1);owner=substr($0,nch+1);sub(/ *$/,"",owner);pid=owner;argv=owner;sub(/\/.*/,"",pid);sub(/[^\/]*\//,"",argv);printf "%s\n%s\033[31m[\033[0m%s\033[31m/\033[0m%s\033[31m]\033[0m\n", $0, chrome, pid, argv;argv=substr(argv,1,8);if (argv~/^ssh/){next};if (owner!~/^[0-9]*\//){next};l1=length(pid);l2=length(argv);ln=(l1>l2)?l1:l2;cmd=sprintf("%s %-" ln "s",cmd,pid);meta=sprintf("%s \033[7m%-" ln "s\033[27m",meta,argv)} /[^a-zA-Z0-9]PID\/Program name *$/ {on=1;sub(/PID\/Program name *$/,"");nch=length($0)} END {print;print cmd;print meta}' kill some of the really bad ones # for svc in vsftpd bind9 rpcbind apache2 nginx gotto; do service $svc stop; done repeat the listening check to see if you missed some determine the hdd name in the rescue environment by using one or more of the following commands (it will be the largest drive connected) # lsblk # blkid # fdisk -l /dev/sd* /dev/nvme* all remaining steps assume that the disk is /dev/sda, adjust accordingly every time you enter the rescue environment, ensure that nothing is attempting to use the disk (this includes mdadm, lvm, systemd automount, etc) try to stop some common offenders most of these will fail, this is intended # service udev stop # service mdadm stop # lvscan | awk -F\' '/ACTIVE/ {print $2}' | while IFS= read -r x; do echo "stopping $x"; umount "$x"; lvchange -an "$x"; done # vgchange -an # dmsetup remove_all # mdadm --stop /dev/md* # mdadm --remove /dev/md* ensure the disk is not in use before continuing (this command should have no output) # mount | grep -E dev/'(sda|nvme|dm|mapper)' if you have time to spare, stresstest the server HDD by filling it with random data expect speeds up to 100 MB/s depending on your hard drive this may take hours so please use tmux or screen collect the current SMART info from the disk # smartctl -x /dev/sda > smart1 fill the disk with junk # openssl enc -aes-256-ctr -pass pass:hunter2 -nosalt < /dev/zero | dd iflag=fullblock of=/dev/sda bs=512k collect SMART again # smartctl -x /dev/sda > smart2 check to see if anything looks spoopy # diff -NarU0 smart{1,2} if you are extra paranoid, you can check if all the bits were written correctly since the "random" data was seeded with a known salt (hunter2) expect speeds up to 100 MB/s depending on your hard drive this may take hours so please use tmux or screen # mkfifo fa fb # openssl enc -aes-256-ctr -pass pass:hunter2 -nosalt < /dev/zero > fa & # dd if=/dev/sda bs=512k of=fb & # cmp fa fb disk is probably ok if cmp prints something about EOF, for example cmp: EOF on fb after byte , in line disk is definitely bad if cmp says something about difference now that the server is in some rescue environment, gather the information we need during the vm install 1) collect the GATEWAY and active NIC by running # ip route | grep default take note of the GATEWAY and the NIC (annotated example output below) default via 12.34.56.1 dev eth0 onlink |GATEWAY|| NIC| 2) collect your PREFIX (12.34.56.78/24) by running # ip addr | grep $NIC take note of your PREFIX which includes the /nn (annotated example output below) inet 12.34.56.78/24 brd 12.34.56.255 scope global eth0 ||||PREFIX|||| so all we need is GATEWAY = 12.34.56.1 PREFIX = 12.34.56.78/24 leave the server sitting in rescue mode, we will eventually replace its HDD contents with the VM below ON YOUR WORKSTATION ################### download the latest extended alpine iso from https://mirror.leaseweb.com/alpine/latest-stable/releases/x86_64/ for example https://mirror.leaseweb.com/alpine/latest-stable/releases/x86_64/alpine-extended-3.8.2-x86_64.iso create a new virtualbox VM Name alpine Type Linux 4.x (64-bit) -OR- Ubuntu (64-bit) Memory 1024 MB or more Harddisk VDI Dynamically allocated 2 GB modify the VM settings Storage » Controller: SATA [x] Use Host I/O Cache Storage » Controller: SATA » alpine.vdi » Solid-state Drive set yes/no to match your server (if it has an SSD or not) Storage » Controller: IDE » Optical Drive attach the alpine iso boot the VM, login as root install any additional packages before continuing # apk add nano vim prepare the install command in a variable # c="ROOTFS=btrfs SWAP_SIZE=0 MKFS_OPTS_ROOT=-f setup-alpine" check for typos in the command # echo "$c" | md5sum which should print ab79793ab37a8222b452cffd39eca323 start installation if that looks roughly correct # eval $c perform the manual installation steps Select keyboard layout [none]: us Select variant []: us-altgr-intl Enter system hostname [localhost]: memes Which one do you want to initialize? [eth0] Ip address for eth0? [dhcp] Gateway? [none] Manual network configuration? [no] DNS domain name? [] DNS nameserver(s)? [] 1.1.1.1 New password: Timezone [UTC] HTTP/FTP proxy URL? [none] Enter mirror number [f]: done Which SSH serveR? [openssh] Which NTP client to run? [chrony] Which disk(s) to use? [none] sda How would you like to use it? [?] sys Erase above disks and continue? [y/N]: y terminate the vm by running # poweroff open vm settings remove the alpine iso reboot the VM login with root and your password, then modify sshd settings to allow logging in as root remotely # nano /etc/ssh/sshd_config locate #PermitRootLogin duplicate the line with CTRL-K, CTRL-U, CTRL-U modify the duplicated line so it says PermitRootLogin yes terminate the vm by running # poweroff collect the sha256 checksum of the vm image: # cd ~/"VirtualBox VMs/alpine" # sha256sum alpine.vdi compress the vm image if the server rescue environment has the "xz" command # xz -cze9T4 < alpine.vdi > alpine.vdi.xz if it doesn't # gzip -c < alpine.vdi > alpine.vdi.gz transfer the vm image to the server # scp alpine.vdi.* root@12.34.56.78: ON THE SERVER (RESCUE MODE) ########################### write the vdi file to the server hdd if the server rescue environment has the "xz" command # xz -d < alpine.vdi.xz > alpine.vdi if it doesn't # gzip -d < alpine.vdi.gz > alpine.vdi write the vdi file to the server disk # qemu-img convert -O raw alpine.vdi /dev/sda if qemu-img does not exist, either install it in the rescue environment or use vdi2raw.py instead # wget https://ocv.me/dev/vdi2raw.py # curl -O https://ocv.me/dev/vdi2raw.py # chmod 755 vdi2raw.py # ./vdi2raw.py alpine.vdi > /dev/sda ensure that all the writes have completed # sync reboot the server in normal mode (non-rescue-mode), there is no need to perform a clean shutdown ssh into the server # ssh root@12.34.56.78 you did it ON THE SERVER (ALPINE) ###################### now that we booted into alpine and left the rescue environment, the hdd may have changed to a different name the new name will be the first line printed when you run # ls -1 /dev | grep -E "^sd|nvme[0-9]+n" fix the apk repositories file so you can install packages # echo https://mirror.leaseweb.com/alpine/v3.8/main > /etc/apk/repositories # echo https://mirror.leaseweb.com/alpine/v3.8/community >> /etc/apk/repositories install some important packages # apk update # apk add bash util-linux e2fsprogs-extra smartmontools if you want the unencrypted os parittion to fill the entire hdd (values in [brackets] are default, just hit enter) # bash # hash -r # fdisk /dev/sda p d [2] Partition number (1,2, default 2): n [p] Partition type (p=primary, e=extended) [2] Partition number (2-4, default 2): [131072] ...but double check! (set it the same as old sda2 start) [whatever] Last sector, +sectors or +size n "Partition #2 contains a ext4 signature. Do you want to remove the signature?" w and then because partprobe may not work (?) # reboot finally expand the filesystem # btrfs filesystem resize max / # btrfs filesystem sync / # sync # reboot install some important packages (improves performance and compatibility) # apk add bash bzip2 coreutils curl file findutils gawk grep gzip htop less lsof man mc mosh nano ncdu ncurses ncurses-terminfo ncurses-terminfo-base netcat-openbsd nmap-ncat p7zip pigz pv python3 screen smartmontools socat strace tar tmux unrar unzip util-linux vim wget xz zip install some additional nice-to-haves # apk add alpine-sdk axel e2fsprogs e2fsprogs-extra ffmpeg flac gdb gnuplot icecast iftop imagemagick lame libtool mariadb mariadb-client mkvtoolnix mpg123 murmur mutagen nasm ncurses-dev nginx nload nmap nmap-scripts php7-fpm python2 rsync rtorrent sudo vorbis-tools xdelta3