By /robex/, July 2018. Back to Articles and Guides
Trim video in FFmpeg without reencoding (parameter order matters):
# -ss: start time # -to: end time # for length, use -t [duration] instead ffmpeg -ss HH:MM:SS -i input.mp4 -to HH:MM:SS -c copy output.mp4
Download part of Twitch stream VOD:
# -ss: start time # -t: duration ffmpeg -ss HH:MM:SS -i $(youtube-dl -g https://www.twitch.tv/videos/1111111111) -t HH:MM:SS -c copy .\clip.mp4
Download part of YouTube video with yt-dlp:
yt-dlp -f 22 --download-sections "*00:00-10:00" https://youtu.be/aaaaaaaaaaa
Limit max resolution to 720p with yt-dlp:
yt-dlp -f 'bv[height<=720]+ba/b[height<=720]' url
Combine an image and an audio file in FFmpeg, for it to be playable in Firefox:
ffmpeg -loop 1 -i image.png -i audio.mp3 -c:v libx264 -tune stillimage -c:a libmp3lame -b:a 192k -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -pix_fmt yuv420p -shortest out.mp4
Resize multiple images using ImageMagick:
convert '*.jpg[640x480]' resized%03d.jpg
Convert multiple wav files to mp3 (bash + lame):
for i in *.wav; do [ -f "$i" ] || break newf=$(echo $i | sed "s/.wav/.mp3/") lame "$i" "$newf" done
Convert multiple wav files to opus (bash + opus-tools):
for i in *.wav; do [ -f "$i" ] || break newf=../opus/$(echo $i | sed "s/.wav/.opus/") opusenc --bitrate 160 "$i" "$newf" done
Convert multiple wav files to mp3 (powershell + FFmpeg):
Get-ChildItem . -Filter *.wav |
Foreach-Object {
$newf = $_.BaseName + ".mp3"
ffmpeg -i $_.FullName $newf
}
Add multiple URLs to push at the same time in git (do this for all URLs):
git remote set-url --add --push origin git@domain:repo
Mass-update GitHub token in Gitea mirrors:
cd /opt/docker/gitea/git/repositories/ # test grep -rlZ ghp_oldtoken . | xargs -0 sed 's/ghp_oldtoken/ghp_newtoken/' # apply grep -rlZ ghp_oldtoken . | xargs -0 sed -i 's/ghp_oldtoken/ghp_newtoken/'
Mirror bare repository and update it whenever you need:
git clone --mirror git@domain:repo # update it git remote update
Cache credentials for 15 minutes on your machine:
git config --global credential.helper cache
Script to update all repos under directory repositories:
#!/bin/bash for f in repositories/*/; do # redundant with */, checks if f is not directory if [ -d $"f" ]; then # paretheses create a subshell, so current working directory doesnt change ( cd "$f" && git remote update ) fi done
Windows: add resource file (icon in this case) and compile with fasm:
# in file rsrc.rc: 100 icon discardable "ico.ico" # then, in cmd: \masm32\bin\rc.exe rsrc.rc \masm32\bin\cvtres.exe rsrc.RES # in asm source code section '.rsrc' resource from 'rsrc.RES' data readable
Then, when linking, add rsrc.obj or rsrc.res. Note: for some reason in Windows 7, only linking with rsrc.obj shows icon in taskbar.
Mirror data partition with robocopy:
# usage: robocopy [src] [dst] # /MIR mirror # /V verbose (print each file) # /XD exclude directory # /R:3 try reads only 3 times # /W:15 try writes only 15 times robocopy D:\ E:\ /MIR /XD 'D:\$RECYCLE.BIN\' /XD 'D:\System Volume Information' /V /R:3 /W:15
Print names of files with the same size:
get-childitem $args[0] -rec -force -file | group-object -property length | where-object count -gt 1 | sort -property count -descending | format-table -wrap -autosize
Run .bat file as administrator:
# add at start of script
set "params=%*"
cd /d "%~dp0" && ( if exist "%temp%\getadmin.vbs" del "%temp%\getadmin.vbs" ) && fsutil dirty query %systemdrive% 1>nul 2>nul || ( echo Set UAC = CreateObject^("Shell.Application"^) : UAC.ShellExecute "cmd.exe", "/k cd ""%~sdp0"" && %~s0 %params%", "", "runas", 1 >> "%temp%\getadmin.vbs" && "%temp%\getadmin.vbs" && exit /B )
Rename files in current directory (non-recursively) with invalid Windows characters in filename, replacing them by _:
# test first rename -n 's/[?<>\\:*|\"]/_/g' * # actual run rename 's/[?<>\\:*|\"]/_/g' *
grep on stderr:
./script 2> >(grep 'abcd')
Remove youtube id from youtube-dl filenames (e.g. file-6b3kZ3q2i.mp4 -> file.mp4):
rename -n 's/-[^.-]+(?=\.)//' *.* # where: - # match a hyphen [^.-]+ # match 1 or more of any char that is not hyphen or dot (?=\.) # lookahead to assert that next char is dot
Sort human-readable du output:
du -h [path] | sort -h # reversed (largest first) du -h [path] | sort -hr
Escape HTML characters:
recode ..html < file.txt
Print binary diff in linux:
cmp -l hex.bin hex2.bin | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
# C-struct style (print offset, byte that is different in hex2.bin)
cmp -l hex.bin hex2.bin | gawk '{printf "{0x%08X, 0x%08X},\n", $1, strtonum(0$3)}'
Open multiple files in tabs in vim (use :Tabe [files/regex]):
" Open multiple tabs at once fun! OpenMultipleTabs(pattern_list) for p in a:pattern_list for c in glob(l:p, 0, 1) execute 'tabedit ' . l:c endfor endfor endfun command! -bar -bang -nargs=+ -complete=file Tabe call OpenMultipleTabs([<f-args>])
Bash script to add a header to multiple files:
#!/bin/bash
display_usage() {
echo "Usage:"
echo -e "./sethead [header file] [files] \n"
}
if [ $# -lt 2 ]
then
display_usage
exit 1
fi
header=$1
for file in ${@:2}; do
cat $header $file > "$file"tmp
mv "$file"tmp $file
done
Create logical backup of mysql database:
mysqldump -h [host] -u [user] -p [dbname] > file.sql # to restore it, just load it like a normal db: mysql -h [host] -u [user] -p < file.sql
Dump query result to file, keeping ASCII table format:
mysql -e 'select * from [table]' -t -u [user] -h [host] -p [dbname] > query.txt
Check HDD for bad blocks with a single pass (pattern 0x55) destructive test, and write output to badblocks.txt (WARNING: THIS WILL DESTROY ALL DATA ON THE DISK!):
sudo badblocks -wsv -t 85 -c 65536 -o badblocks.txt /dev/sda
Clone disk/partition ignoring bad sectors (without trying to re-read bad sectors):
sudo ddrescue -d -f -n /dev/input /dev/output rescue.log
Long SMART test:
sudo smartctl -t long /dev/sdx # check test progress with sudo smartctl -a /dev/sdx
On ext4 partitions, don't reserve 5% of the filesystem space for root (use only on non-system partitions):
sudo tune2fs -m 0 /dev/sdxy
Apply user quotas in linux filesystem:
# enable quota options in /etc/fstab /dev/uuid /mount ext4 defaults,usrquota 0 0 # remount partition mount -vo remount /mount # create quota index and turn quota on quotacheck -cum /mount quotaon -v /mount # set quotas for a user (set soft/hard to number of usable 1KB blocks) EDITOR=vi edquota user
Create RAID1 array:
# create array mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb # wait for the array to finish syncing, check progress with 'cat /proc/mdstat' # create filesystem and mount array (i.e. ext4) mkfs.ext4 -F /dev/md0 # save array layout mdadm --detail --scan | tee -a /etc/mdadm/mdadm.conf update-initramfs -u # auto-mount on startup (append line to /etc/fstab) /dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0
Rebuild failed RAID1 array:
# get serial number of failed disk for d in /dev/sd?; do echo $d; hdparm -I $d | grep Number; done # remove failed disk mdadm --manage /dev/md0 --fail /dev/sdx mdadm --manage /dev/md0 --remove /dev/sdx # check status with cat /proc/mdstat # replicate partitioning in the new drive (in this example, copy sda partitions to sdb) sfdisk -d /dev/sda | sfdisk /dev/sdb # add the new drive to the array mdadm --manage /dev/md0 --add /dev/sdx # check synchronization status with the same command as above
Encrypt directory in ext4 filesystem with fscrypt:
# enable encrypt feature tune2fs -O encrypt /dev/sdxy # setup fscrypt (NEVER delete .fscrypt directory or all encrypted data will be LOST) fscrypt setup fscrypt setup /mount/point # encrypt a directory fscrypt encrypt private/ # unlock a directory fscrypt unlock private/ # lock a directory # old versions of fscrypt, requires root and locks ALL directories in the filesystem fscrypt /mount/point --user=user # new versions, doesnt require root fscrypt lock private/
Email alert on failed RAID1 drive (make sure you can send mail with sendmail, check guide in "E-mail" section):
# append to /etc/mdadm/mdadm.conf MAILADDR [email protected] # test run (^C after email received) sudo mdadm --monitor /dev/md0 --test
Create SMB only user:
sudo adduser --no-create-home --disabled-password --disabled-login sambausername sudo smbpasswd -a sambausername service smbd force-reload
Share linux folder with SMB:
apt install samba # create SMB user (see above) # append to /etc/samba/smb.conf [folder_name] path = /path/to/folder valid users = john, mike read only = no # restart service and test service smbd force-reload testparm
Permanently map SMB drive in Windows:
net use Z: \\mynas\robex
Setup postfix Gmail relay (send email from Gmail account with sendmail command):
sudo apt-get install libsasl2-modules postfix # during postfix install select "Internet Site" and type FQDN (doesnt have to be a real one) # check that the following parameter matches the previous FQDN in /etc/postfix/main.cf myhostname = fqdn.example.com # add the following line to /etc/postfix/sasl/sasl_passwd (create the file if it doesnt exist) [smtp.gmail.com]:587 [email protected]:password # create hashmap (should generate file sasl_passwd.db) sudo postmap /etc/postfix/sasl/sasl_passwd # change permissions to files containing plaintext passwd sudo chmod 0600 /etc/postfix/sasl/sasl_passwd* # configure relay in /etc/postfix/main.cf, add or modify the following params relayhost = [smtp.gmail.com]:587 # Enable SASL authentication smtp_sasl_auth_enable = yes # Disallow methods that allow anonymous authentication smtp_sasl_security_options = noanonymous # Location of sasl_passwd smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd # Enable STARTTLS encryption smtp_tls_security_level = encrypt # Location of CA certificates smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt # restart postfix sudo systemctl restart postfix # note: remember that you must disable 2factor auth and enable less secure apps on gmail security settings # (for some stupid reason gmail seems to disable this automatically every so often) # test sending an email: sendmail [email protected] From: [email protected] Subject: Test mail This is a test email .
Setup postfix GMX relay (send email from GMX account with sendmail command):
sudo apt-get install libsasl2-modules postfix # in /etc/postfix/main.cf myhostname = gmx.es myorigin = $myhostname # add the following line to /etc/postfix/sasl/sasl_passwd (create the file if it doesnt exist) [mail.gmx.es]:587 [email protected]:password # create hashmap (should generate file sasl_passwd.db) sudo postmap /etc/postfix/sasl/sasl_passwd # change permissions to files containing plaintext passwd sudo chmod 0600 /etc/postfix/sasl/sasl_passwd* # configure relay in /etc/postfix/main.cf, add or modify the following params relayhost = [mail.gmx.es]:587 # Enable SASL authentication smtp_sasl_auth_enable = yes # Disallow methods that allow anonymous authentication smtp_sasl_security_options = noanonymous # Location of sasl_passwd smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd # Enable STARTTLS encryption smtp_tls_security_level = encrypt # Location of CA certificates smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt # force emails to be sent with your gmx account username, so they dont get blocked # in /etc/postfix/main.cf smtp_generic_maps = hash:/etc/postfix/generic # in /etc/postfix/generic (example to redirect root to myuser) [email protected] [email protected] # lastly, create hashmap sudo postmap /etc/postfix/generic # restart postfix sudo systemctl restart postfix
Check postfix queue:
mailq
Delete postfix queue:
postsuper -d ALL
Make PIA socks5 proxy work with JDownloader2:
# in cfg folder, find this file org.jdownloader.settings.InternetConnectionSettings.customproxylist.json # change all occurrences to true except for the first one (null) "resolveHostName":false
Enable tun/tap for lxc containers (necessary for openvpn):
sudo vi /var/lib/lxc/<container-name>/config
# add the following lines
lxc.cgroup2.devices.allow = c 10:200 rwm
lxc.hook.autodev = sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"
sudo lxc-update-config -c /var/lib/lxc/<container-name>/config
Create manual bridge for lxc containers:
# in /etc/network/interfaces
auto br0
iface br0 inet static
bridge_ports enp2s0f0 # main interface here
bridge_fd 0
address 192.168.x.x # address of this host
network 192.168.x.0
broadcast 192.168.x.255
netmask 255.255.255.0
gateway 192.168.x.x
dns-nameservers 192.168.x.x
# restart networking
sudo /etc/init.d/networking restart
# in /etc/default/lxc-net
USE_LXC_BRIDGE="false"
# in /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1
Test iptables rules to avoid locking yourself out (they will revert in 10 seconds if not confirmed by user):
sudo iptables-apply iptables_ssh.rules # example content of iptables_ssh.rules: *filter -I INPUT -p tcp --dport 22 -s 1.1.1.1 -j ACCEPT -I INPUT -p tcp --dport 22 -s 192.168.0.0/16 -j ACCEPT -A INPUT -p tcp --dport 22 -j REJECT COMMIT
Change display idle dim time on Ubuntu MATE:
gsettings set org.mate.power-manager idle-dim-time <seconds>
Backup NetLimiter 4 rules:
net stop nlsvc # save %ProgramData%\Locktime\4\nl_settings.xml net start nlsvc
Update teamspeak3 server:
# download latest version at https://www.teamspeak.com/en/downloads/#server wget https://files.teamspeak-services.com/releases/server/3.13.3/teamspeak3-server_linux_amd64-3.13.3.tar.bz2 # in your teamspeak folder: ./ts3server_startscript.sh stop # backup database just in case cp ts3server.sqlitedb /tmp/ # extract new server and overwrite all files tar xfv teamspeak3-server_linux_amd64-3.13.3.tar.bz2 cp -r teamspeak3-server_linux_amd64/* . # this file should already exist, but in case it doesnt, create it touch .ts3server_license_accepted # start the server again ./ts3server_startscript.sh start
Mirror website with wget:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://example.org
Schedule command to run as root at a specific date and time with at:
sudo at 15:00:00 2020-03-22 # introduce whatever command here, this is just an example to run a script at> sh script.sh # press control+d
Pipe output of command to X clipboard:
echo 'example' | xclip -selection clipboard
Create standalone certificate with certbot:
sudo certbot certonly --standalone -d robex.ga
Record framebuffer:
sudo ffmpeg -f fbdev -i /dev/fb0 -r 24 -c:v libx264 -b:v 500k output.mp4
Set dvorak-programmer as layout for tty/login manager in Ubuntu:
sudo vi /etc/default/keyboard XKBMODEL="SKIP" XKBLAYOUT="us" XKBVARIANT="dvp" # for normal dvorak, put "dvorak" here XKBOPTIONS="" BACKSPACE="guess"
Enable auto-updates on Debian:
apt-get install -y unattended-upgrades apt-listchanges echo 'Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";' >> /etc/apt/apt.conf.d/52unattended-upgrades-local echo 'Unattended-Upgrade::Automatic-Reboot "false";' >> /etc/apt/apt.conf.d/52unattended-upgrades-local echo 'APT::Periodic::Update-Package-Lists "1";' >> /etc/apt/apt.conf.d/20auto-upgrades echo 'APT::Periodic::Unattended-Upgrade "1";' >> /etc/apt/apt.conf.d/20auto-upgrades