How to setup a VNC Server on Linux?

Display-Server

How to configure and secure a VNC server on Linux with TigerVNC? (on a screen less server or a classic machine)

Since the VNC Server (TigerVNC) configuration is the same on most Linux distributions and only the installation method differ, this wiki is targeting: OpenSUSE, Fedora, CentOS, RHEL, Debian, Mageia, Void Linux, Arch Linux, Manjaro and FreeBSD (in order to be useful for more people)

Installing a VNC Server

On Linux (on a classic machine or a screen less server) there are multiple (opensource) possibility for a VNC server such as TightVNC, TigerVNC and TurboVNC (this is a non exhaustive list, this guide will be using the native version of TigerVNC):

We first need a desktop (E.g. XFCE or KDE):

# OpenSUSE (XFCE)
zypper in -t pattern xfce

# OpenSUSE (KDE)
zypper install -t pattern kde kde_plasma

# Fedora/OpenSUSE (XFCE)
dnf groupinstall -y "Xfce Desktop"

# Fedora/OpenSUSE (KDE)
dnf -y group install "KDE Plasma Workspaces" 

# CentOS/RHEL (Gnome)
dnf -y group install "Server with GUI"

# CentOS/RHEL (XFCE)
dnf --enablerepo=epel group -y install "Xfce" "base-x" 

# CentOS/RHEL (KDE)
dnf --enablerepo=epel group -y install "KDE Plasma Workspaces"

# CentOS v8 (KDE) 
dnf --enablerepo=epel,PowerTools 
dnf -y group install "KDE Plasma Workspaces" "base-x" 

# Debian (XFCE)
apt install task-xfce-desktop

# Debian (KDE)
apt install task-kde-desktop

# Mageia (XFCE)
dnf install task-xfce

# Mageia (KDE)
dnf install task-plasma5

# FreeBSD (XFCE)
pkg install xfce

# FreeBSD (KDE)
pkg install x11/kde5

# Void Linux (XFCE)
xbps-install -S xfce4

# Void Linux (KDE)
xbps-install -S kde5
# and optionally, kde5-baseapps

# Arch Linux (XFCE)
pacman -S xfce4 xfce4-goodies

# Arch Linux (KDE)
pacman -S plasma-desktop
# or plasma for the full desktop
# pacman -S plasma

# Manjaro (XFCE)
pacman -S xfce4-gtk3 xfce4-goodies xfce4-terminal \
network-manager-applet xfce4-notifyd-gtk3 \
xfce4-whiskermenu-plugin-gtk3 tumbler engrampa

# Manjaro (KDE)
pacman -S plasma kio-extras
# optional kde-applications

Install the TigerVNC X server:

# The package name may change depending on the used distro

# CentOS
yum install tigervnc-server

# Mageia/Fedora/CentOS/RHEL
dnf install tigervnc-server

# ALT Linux
apt install tigervnc-server

# openSUSE DNF
dnf install xorg-x11-Xvnc

# openSUSE
zypper install xorg-x11-Xvnc

# Debian
apt install tigervnc-standalone-server tigervnc-common

# FreeBSD
pkg install tigervnc-server

# Void Linux
xbps-install -S tigervnc

# Arch Linux
pacman -S tigervnc

# Manjaro
pacman -S tigervnc

Setup and configuration

Setup the password (the hashed version will be saved at ~/.vnc/passwd):

vncpasswd

Edit the config file (startup script, executed when the server start) ~/.vnc/xstartup as follow:

#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

exec startxfce4

# XFCE: startxfce4 or xfce4-session
#exec startxfce4
#exec xfce4-session

# KDE: startkde or startplasma-x11
#exec startkde
#exec startplasma-x11

# Gnome: startx
#exec startx

Prepare and/or locate the VNC server’s config file at:

~/.vnc/config or /etc/vnc/config

Adapt VNC server’s settings with the config file… to get the full list of available options we can use Xvnc -help or man Xvnc, here is a config example (also note that on some system like Suse VNC will/may not work if the option geometry is not set):

## Supported server options to pass to vncserver upon invocation can be listed
## in this file. See the following manpages for more: vncserver(1) Xvnc(1).
## Several common ones are shown below. Uncomment and modify to your liking.
##
##
# -------------
#  Xvnc --help
# -------------
##
# Start server : vncserver
# Stop server : vncserver -kill :1
##
###############################################################################

# Only allow connection from local hosts
#localhost

# VNC tcp port
rfbport=5900

# TCP port to listen for HTTP (default=0)
httpport=0

# Directory containing files to serve via HTTP (default=)
httpd=

# Protocols...
#nolisten=UDP
#listen=TCP

# IP settings
useipv4
#useipv6

# Interface, listen on the specified network address (default=all)
#interface=127.0.0.1

# Use protocol version 3.3 for backwards compatibility
protocol3.3=0

# Unix socket access mode (default=384)
#rfbunixmode=384

# Unix socket to listen for RFB protocol (default=)
rfbunixpath=

# Name of VNC desktop
desktop=MyVNC

# Geometry original peppy
geometry=1366x768

# Colors
depth=24

# Sharing with multiple clients
#alwaysshared
nevershared

# Disconnect existing clients if an incoming connection is non-shared. 
# If combined with NeverShared then new connections will be refused while 
# there is a client active
disconnectclients

# Security, specify which security scheme to use (None, VncAuth, Plain,
# TLSNone, TLSVnc, TLSPlain, X509None, X509Vnc, X509Plain) (default=TLSVnc,VncAuth)
securitytypes=TLSVnc,VncAuth

# Path to the key of the X509 certificate in PEM format (default=)
#X509Key=

# Path to the X509 certificate in PEM format (default=)
#X509Cert=

# Set maximum number of clients (power of two)
#maxclients=64

# Terminate after s seconds of user inactivity (default=0)
#maxidletime=0

# Terminate when a client has been connected for s seconds (default0)
#maxconnectiontime=0

# Terminate when no client has been connected for s seconds (default=0)
#maxdisconnectiontime=0

# The number of seconds after which an idle VNC connection will be dropped 
# (zero means no timeout) (default=0)
#idletimeout=0

# Zlib compression level (default=-1)
#zlibLevel=-1

# The maximum number of updates per second sent to each client (default=60)
#framerate=60

# GnuTLS priority string that controls the TLS session’s handshake algorithms.  
# See the GnuTLS manual for possible values. Default is NORMAL.
#GnuTLSPriority=

VNC server can be started/stopped with the following commands; After starting it you can connect to your server with any VNC client to server-ip:used-port(note that you may probably need to open the used port on the firewall)

# Start the server
vncserver

# Stop the server :1
vncserver -kill :1

# Forcing multiple server to stop
killall Xvnc

Security

Basic VNC setup does not use encryption for the exchanged stream, here are 4 commons way of securing a VNC connection:

VNC as a systemd service

VNC server can be used as a Systemd’s service with the following config file /etc/systemd/system/vncserver.service, enabling the service with systemctl enable vncserver.service will make it auto-start when the system boot (this does not apply to Void Linux or Systemd less Linux’s distributions).

# /etc/systemd/system/vncserver.service

[Unit]
Description=TigerVNC Server
After=syslog.target network.target

[Service]
Type=simple
#Type=forking
User=MY-USER
Group=MY-USER-GROUP

#If ran with root
#WorkingDirectory=/root
#PIDFile=/root/.vnc/%H%i.pid

#If ran with any other user
WorkingDirectory=/home/MY-USER
PIDFile=/home/%u/.vnc/%H%i.pid

#Environment is required when using a custom GnuTLS version
#Environment=LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib

ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1
#ExecStart for forking type version
#ExecStart=/usr/bin/vncserver
ExecStart=/usr/bin/vncserver -fg
ExecStop=/usr/bin/vncserver -kill :%i

[Install]
WantedBy=multi-user.target

Generating an X509 certificate

For a simple secure setup this official wiki can be followed, otherwise this is not a light topic for a further understanding of such encryption see the documentation/links for this section. In short the documentation/links indicate that most secure keys as of 2020 are RSA with a high key size (at least 4096 bits) and EdDSA’s Ed25519 implementation which is the EdDSA signature scheme using SHA-512 (SHA-2) and Curve25519 (an elliptic curve offering 128 bits of security). Ed25519 resistance is equivalent to an RSA with a 3072-bits key. (alternatively there is EdDSA’s ED448 which is the EdDSA signature scheme using SHAKE256 (SHA-3) and Curve448; it is equivalent to an RSA with a ~12448-bits key, an updated OpenSSL install is required)

Generate an RSA-4096-Bits/Ed25519/ED448 private key for the CA:
For Ed25519 and ED448 an updated version of OpenSSL and GnuTLS are required (check the next section).

mkdir ~/.vnc/ssl

# Generate an RSA 4096-bits key
openssl genrsa -out ~/.vnc/ssl/ca.private.rsa.4096.key.pem 4096
#
# genrsa is superseded by genpkey (PKCS#1 vs PKCS#8 format), the following command is similar to the previous one.
# openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ~/.vnc/ssl/ca.private.rsa.4096.key.pem

# Or an ED25519 key (equivalent to an RSA with a 3072-bits key, updated openssl required)
openssl genpkey -algorithm ED25519 -out ~/.vnc/ssl/ca.private.eddsa.ed25519.key.pem   

# Or an ED448 key (equivalent to an RSA with a ~12448-bits key, updated openssl required)
openssl genpkey -algorithm ED448 -out ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem

The previous step can be done in a more secure way by protecting the key with a password but TigerVNC does not support that.

Check/View the newly generated key (not required):

openssl pkey -in ~/.vnc/ssl/ca.private.rsa.4096.key.pem -text
# or
openssl pkey -in ~/.vnc/ssl/ca.private.eddsa.ed25519.key.pem -text
# or
openssl pkey -in ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -text

Generate a signing CA, make it valid for 2 years and adding the server IP (this is required, change 88.44.88.33 with your IP):

# RSA 4096-bits 
openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.rsa.4096.key.pem -out ~/.vnc/ssl/ca.sign.rsa.4096.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

#Or ED25519
openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -out ~/.vnc/ssl/ca.sign.eddsa.ed448.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

# Or ED448
openssl req -new -x509 -days 730 -key ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem -out ~/.vnc/ssl/ca.sign.eddsa.ed448.key.pem -subj '/CN=88.44.88.33' -addext "subjectAltName=IP:88.44.88.33"

Update the server configuration to use X509:

# Security, specify which security scheme to use (None, VncAuth, Plain,
# TLSNone, TLSVnc, TLSPlain, X509None, X509Vnc, X509Plain) (default=TLSVnc,VncAuth)
securitytypes=X509Vnc

# Path to the key of the X509 certificate in PEM format (default=)
X509Key=/home/USER/.vnc/ssl/ca.private.eddsa.ed448.key.pem

# Path to the X509 certificate in PEM format (default=)
X509Cert=/home/USER/.vnc/ssl/ca.sign.eddsa.ed448.key.pem

Connecting with TigerVNC Viewer:

Checking the GnuTLS session’s handshake algorithms

TigerVNC is using GnuTLS for the encryption, on the server/client the setting GnuTLSPriority sets the priority string that controls the TLS session’s handshake algorithms (TLS1.0/TLS1.1/TLS1.2/TLS1.3/etc.); supported algorithms can be listed with gnutls-cli --list

We can for instance test the TLS v1.2 support with the following:

vncviewer GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.2 -log='*:stdout:100'

Here is how we can enforce TLS v1.2/v1.3 on the server config file:

# GnuTLS priority string that controls the TLS session’s handshake algorithms.  
# See the GnuTLS manual for possible values. Default is NORMAL.

# Only TLS v1.2
#GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.2

# Only TLS v1.3
GnuTLSPriority=NORMAL:-VERS-ALL:+VERS-TLS1.3

# Verifying if only TLS v1.2/v1.3 policy is working with the following
# vncviewer GnuTLSPriority=NORMAL:-VERS-TLS1.2 -log='*:stdout:100' # v1.2
# vncviewer GnuTLSPriority=NORMAL:-VERS-TLS1.3 -log='*:stdout:100' # v1.3
# This mean use all but v1.2/v1.3 to test if the setting is enforced correctly,
# and thus the connection will be refused for handshake algorithm mismatch.

# Other example of gnutlspriority values (warning, this is just for the syntax)
#GnuTLSPriority=NORMAL:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1:-VERS-TLS1.3
#NORMAL:+SECURE128:-SHA384:-SHA256:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1
#NORMAL:+VERS-TLS1.2:+VERS-TLS1.3:+AES-128-CBC:+RSA:+SHA1:+COMP-NULL

Updating OpenSSL and GnuTLS

OpenSSL and GnuTLS are two main application/library of most Linux system, updating them manually may introduce security issues as they will not be updated automatically anymore, we can limit that side negative effect by using a custom version just for our purpose (TigerVNC).

A newer OpenSSL version is needed to generate a Ed25519/ED448 key, while GnuTLS is needed for vncviewer(client’s viewer) / Xvnc (TigerVNC server) to add support for Ed25519/ED448 algorithms

Check what algorithm is currently supported:

#GnuTLS
gnutls-cli --list | grep EdDSA

#OpenSSL
man -P cat genpkey | grep "Valid built-in algorithm"
openssl list -public-key-algorithms | grep ED

#If EdDSA targeted algorithm is supported there is no need to install from sources

Build, install and use OpenSSL:

wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
tar -xvf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g/
./config no-nextprotoneg no-weak-ssl-ciphers no-ssl3 no-shared -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong enable-tls1_3
make install -j2

# After install, OpenSSL can be used 
# for instance as follow to generate the needed key  
/usr/local/bin/openssl genpkey -algorithm ED448 -out ~/.vnc/ssl/ca.private.eddsa.ed448.key.pem

Build, install and use GnuTLS:

wget https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.14.tar.xz
tar -xvf gnutls-3.6.14.tar.xz
cd gnutls-3.6.14/
./configure --without-tpm --disable-tests --disable-full-test-suite --disable-non-suiteb-curves --disable-ssl2-support
make install -j2

# This is required for Xvnc and vncviewer, after install can be used 
# by setting LD_LIBRARY_PATH before running vncviewer or Xvnc, 
# like the following example

# TigerVNC Server
export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib
vncserver

# TigerVNC Viewer
env LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib:/usr/lib vncviewer

# To check the used version we can use ldd for instance 
ldd /usr/bin/vncviewer

Experiments notes

While using KDE with TigerVNC server; when stopping/starting the server several times, some X applications remain running, probably related to this systemd issue, to fix that the following script can be used to stop the server (can be used on the systemd service config file as well), and ps aux | sort | grep USER-NAME | grep -v '\[' command can be used to check if something is still running after stopping the server.

#cat /usr/bin/vncserver-stop (this is meant for KDE)
 
#!/bin/sh

vncserver -kill :0
vncserver -kill :1
vncserver -kill :2
vncserver -kill :3

killall Xvnc -9
killall kwin_x11 -9
killall startplasma-x11 -9
killall plasma_session -9

killall Xvnc -9
killall kwin_x11 -9
killall startplasma-x11 -9
killall plasma_session -9

After several tests with the viewer, the best speed performances were achieved with vncviewer -QualityLevel=4 -CompressLevel=2 -PreferredEncoding=Raw

If the server needs to be exposed to the internet, note that VNC Server can be easily discovered with an nmap scan (nmap -sV -sC TARGET-IP), VNC protocol need to announce itself and thus without creating a custom version of the server and client there is not much than can be done to obfuscate a VNC server in the case of a global internet exposure.

Documentations

Posted on 2020-06-18 by intika, viewed times and updated on July 24, 2020.

Share this post on:

Related posts:

How does Linux's display work?
How to scale the desktop's resolution?

Contact

Copyright © 2020, LinuxHacks.org