Installing BigBlueButton with Incus

1. Intro

I have a Dedicated Server on hetzner.com and I want to install BigBlueButton. I use Incus on the server, and I want to install BBB in a container.

For installing BBB inside the container, the script bbb-install is recommended, and it seems to work well with a NAT-ed system too. So I don’t need to purchase an extra IP for the BBB container.

The case with an extra IP is covered in this (deprecated) article.

The script that we use also installs a TURN server inside the BBB container itself, so we don’t need to install a TURN server on a separate container or machine (as we have done previously).

2. Preparing the host

To install and setup the Hetzner server, follow the instructions at: https://docker-scripts.gitlab.io/howto/dedicated-rootserver.html

3. Creating the container

The latest stable version of BBB requires ubuntu:20.04.

incus launch images:ubuntu/20.04 bbb \
    -c security.nesting=true \
    -c security.syscalls.intercept.mknod=true \
    -c security.syscalls.intercept.setxattr=true

For more details look at https://docker-scripts.gitlab.io/howto/create-incus-container.html. Make sure to set a fixed IP to this container (for example 10.210.64.201), and to install Docker.

4. Networking

The connection of our container to the network goes through the host, which serves as a gateway. So, the BigBlueButton server is behind NAT. For this reason, we need to forward some TCP and UDP ports to it. This is easier if the container has a fixed IP (rather then a dynamic one, obtained from DHCP).

4.1. Set a fixed IP

Network configuration on ubuntu is managed by netplan.

incus exec bbb -- bash

ip address
ip route

rm /etc/netplan/*.yaml
cat <<EOF > /etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses:
        - 10.210.64.201/8
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]
      routes:
        - to: default
          via: 10.210.64.1
EOF

netplan apply

ip address
ip route
ping 8.8.8.8

4.2. Forward ports

We can use the command incus network forward to forward these ports to the internal IP of the BBB container:

HOST_IP=10.11.12.13           # the public IP of the host
CONTAINER_IP=10.210.64.201

incus network forward create incusbr0 $HOST_IP
incus network forward list incusbr0

incus network forward port add incusbr0 \
    $HOST_IP tcp 3478 \
    $CONTAINER_IP
incus network forward port add incusbr0 \
    $HOST_IP udp 3478,50001-65535 \
    $CONTAINER_IP
incus network forward show incusbr0 $HOST_IP

Test port forwarding

We can use netcat to test that ports are forwarded correctly. On the server run:

incus exec bbb -- nc -u -l 51000

Outside the server run:

nc -u bbb.example.org 51000

We are assuming that bbb.example.org is resolved to the external IP of the server.

Every line that is typed outside the server should be displayed inside the server.

4.3. Forward the TCP ports 80 and 443

Forwarding these two ports is a bit more complex and cannot be done with the same method that was used above for the UDP ports. This is because these ports need to be used by other applications as well, beside BBB. We need to forward these ports to different applications or containers, based on the domain that is being used. We can use sniproxy for this: https://docker-scripts.gitlab.io/howto/create-incus-container.html

Make sure that the configuration file etc/sniproxy.conf looks like this:

table {
    # . . . . .

    bbb.example.org 10.210.64.201

    # . . . . .
}

We are using 10.210.64.201, which is the fixed IP of the bbb container.

5. Installing BBB inside the container

5.1. Run the installation script

Before running the installation script, add first this line at /etc/hosts:

10.11.12.13  bbb.example.org

10.11.12.13 is the external/public IP of the server.

incus exec bbb -- bash

echo "10.11.12.13  bbb.example.org" >> /etc/hosts

apt install -y wget gnupg2
wget http://ubuntu.bigbluebutton.org/repo/bigbluebutton.asc \
    -O- | apt-key add -

base_url="https://raw.githubusercontent.com/bigbluebutton"
wget -q $base_url/bbb-install/v2.7.x-release/bbb-install.sh

chmod +x bbb-install.sh
./bbb-install.sh

Create the customization script /etc/bigbluebutton/bbb-conf/apply-config.sh:

mkdir -p /etc/bigbluebutton/bbb-conf/

cat <<'_EOF_' > /etc/bigbluebutton/bbb-conf/apply-config.sh
#!/bin/bash

# Enable playback of recordings on iOS
sed -i /usr/local/bigbluebutton/core/scripts/presentation.yml \
    -e 's/^# - mp4/- mp4/'

### Cusomize UDP ports
### See: https://docs.bigbluebutton.org/administration/customize/#change-udp-ports

# Use ports 50001-55000 for FreeSWITCH
sed -i /opt/freeswitch/etc/freeswitch/autoload_configs/switch.conf.xml \
    -e '/rtp-start-port/ s/value=".*"/value="50001"/' \
    -e '/rtp-end-port/ s/value=".*"/value="55000"/'

# Use ports 55001-60000 for Kurento
sed -i /etc/kurento/modules/kurento/BaseRtpEndpoint.conf.ini \
    -e '/^minPort=/ c minPort=55001' \
    -e '/^maxPort=/ c maxPort=60000'

# Use ports 55001-60000 for mediasoup
sed -i /etc/bigbluebutton/bbb-webrtc-sfu/production.yml \
    -e '/worker:/,$ d'
cat <<EOF >> /etc/bigbluebutton/bbb-webrtc-sfu/production.yml
  worker:
    rtcMinPort: 55001
    rtcMaxPort: 60000
EOF

# Use ports 55001-60000 for bbb-webrtc-recorder
cat <<EOF > /etc/bigbluebutton/bbb-webrtc-recorder.yml
webrtc:
  rtcMinPort: 55001
  rtcMaxPort: 60000
EOF

# Use ports 60001-65535 for the TURN server
sed -i /etc/turnserver.conf \
    -e '/^min-port/ c min-port=60001' \
    -e '/^max-port/ c max-port=65535'
_EOF_

chmod +x /etc/bigbluebutton/bbb-conf/apply-config.sh

This script will be executed automatically by bbb-install.sh.

Let’s also create the script update.sh, and run it:

cat <<'EOF' > update.sh
#!/bin/bash -x

apt update
apt -y upgrade

cd $(dirname $0)

#base_url="https://raw.githubusercontent.com/bigbluebutton"
#wget -q $base_url/bbb-install/v2.7.x-release/bbb-install.sh
#chmod +x bbb-install.sh

./bbb-install.sh \
    -v focal-270 \
    -s bbb.example.org \
    -e info@example.org \
    -t lti_key:lti_secret \
    -g -k
EOF

lti_key=$(tr -cd '[:alnum:]' < /dev/urandom | fold -w20 | head -n1)
lti_secret=$(tr -cd '[:alnum:]' < /dev/urandom | fold -w20 | head -n1)
sed -i update.sh \
    -e 's/lti_key/$lti_key/' \
    -e 's/lti_secret/$lti_secret/'

chmod +x update.sh
./update.sh

Sometimes it is advisable to run ./update.sh a second time.

5.2. Setup email notifications

To be able to send email notifications, we need to set the SMTP variables on the configuration file greenlight-v3/.env.

If you have installed a simple SMTP server (as described on https://docker-scripts.gitlab.io/howto/disable-ipv6.html) you can set them like this:

SMTP_SENDER_EMAIL=noreply@example.org
SMTP_SENDER_NAME=BBB
SMTP_SERVER=smtp.example.org
SMTP_PORT=25
SMTP_DOMAIN=example.org

Leave the rest of the SMTP settings commented because they are not needed for this case.

For the settings to be applied, we need to rebuild the containers:

cd greenlight-v3/
docker-compose down
docker-compose up -d

Do a quick check with:

docker exec -it greenlight-v3 bundle exec rake configuration:check

For further checking (if needed), try to use swaks:

apt install swaks
swaks --from notify@example.org --to info@example.org -tlso

If the simple SMTP server is installed in the same host as the container bbb, there is a problem, because Incus containers by default cannot access the host. So, we cannot access the port 25 on the host, that is needed to send emails to the SMTP server.

We need to open this port for the BBB container, and we can do it like this:

incus config device add \
    bbb smtp25 proxy \
    listen=tcp:0.0.0.0:25 \
    connect=tcp:0.0.0.0:25 \
    bind=container

incus config device show bbb

5.3. Add admins and users

docker exec greenlight-v3 \
    bundle exec rake \
    admin:create["BBB Admin","admin@example.org","qwerty"]

docker exec greenlight-v3 \
    bundle exec rake \
    user:create["User Name","email@example.org","asdfgh"]

5.4. Enable video format

By default, the recorded videos can be viewed only in the "presentation" format. It is possible to enable other formats as well, like "video" and "screenshare". It can be done like this:

apt install -y \
    bbb-playback-screenshare \
    bbb-playback-video

mkdir -p /etc/bigbluebutton/recording
cat << _EOF_ > /etc/bigbluebutton/recording/recording.yml
steps:
  archive: "sanity"
  sanity: "captions"
  captions:
    - "process:presentation"
    - "process:video"
    - "process:screenshare"
  "process:presentation": "publish:presentation"
  "process:video": "publish:video"
  "process:screenshare": "publish:screenshare"
_EOF_

systemctl restart bbb-rap-resque-worker.service

After this, it is a good idea to reinstall/update the BBB installation (with ./update.sh).

Then, it is also possible to regenerate all the enabled formats for the recorded videos with bbb-record --rebuildall.

Rebuilding all the recoded meeting usually takes a very long time.

6. Appendices

6.1. Using an external storage

If you use are using a Hetzner Dedicated Root Server, as suggested at the beginning of this article, there should plenty of disk space for storing recorded BBB sessions. But if you need even more disk space, you can get a Storage Box and use it for storing the data.

6.1.1. Access the storagebox with SSH

First of all, activate the SSH service of your Storage Box within Robot.

Then follow these instructions to access the Storage Box with SSH keys:

Generate an SSH key pair:

cd ~
mkdir -p storagebox
cd storagebox/

ssh-keygen -q -N '' -f key1
ls key1*

Upload the public key to the Storage Box server:

cat key1.pub \
    | ssh -p23 uXXXXXX@uXXXXXX.your-storagebox.de install-ssh-key

### test it
ssh uXXXXXX@uXXXXXX.your-storagebox.de -p23 -i $(pwd)/key1

Create an SSH config entry:

cat << _EOF_ >> ~/.ssh/config
Host storagebox
    HostName uXXXXXX.your-storagebox.de
    User uXXXXXX
    Port 23
    IdentityFile /root/storagebox/key1
_EOF_

chmod 600 ~/.ssh/config
ls -l ~/.ssh/config
cat ~/.ssh/config

### test it
ssh storagebox

6.1.2. Make a backup of the BBB data

Let’s backup the content of the directory /var/bigbluebutton/ inside the container:

incus shell bbb
bbb-conf --stop

mv /var/bigbluebutton /var/bigbluebutton-1
mkdir -p /var/bigbluebutton
chown bigbluebutton: /var/bigbluebutton
ls -al /var/bigbluebutton

exit

We stopped the BBB services and moved the data directory to /var/bigbluebutton-1.

6.1.3. Mount a host dir to the BBB container

We want a directory on the host system to be mounted on the directory /var/bigbluebutton/ inside the container (because this is directory that contains the BBB data — meeting recordings, etc). Let’s say that we will use the directory /mnt/bbb/ (on the host) for this purpose.

Let’s mount the /mnt/bbb directory by adding it as a disk device to the container:

mkdir -p /mnt/bbb

incus config device add bbb var_bigbluebutton disk \
    source=/mnt/bbb \
    path=/var/bigbluebutton
incus config device show bbb

We have named this device var_bigbluebutton.

By default, the directory is mounted read-only inside the container, so we cannot write in it from inside the container. We have to make it writable (this article explains more about how to do it).
```
cat /etc/subuid
cat /etc/subgid
id

echo "root:0:1" | tee -a /etc/subuid /etc/subgid
cat /etc/{subuid,subgid}

incus config set bbb raw.idmap "both 0 0"
incus config get bbb raw.idmap

incus restart bbb
incus exec bbb -- bbb-conf --stop
```
Remapping container filesystem may take a while (during restart).

Let’s also give the right ownership to the mounted directory:

incus exec bbb -- ls -al /var/bigbluebutton
incus exec bbb -- chown bigbluebutton: /var/bigbluebutton
incus exec bbb -- ls -al /var/bigbluebutton

6.1.4. Use SSHFS to mount the storage box to /mnt/bbb

First let’s create a directory on the storagebox:

ssh storagebox "mkdir -p storage/bbb.example.org/data"
ssh storagebox "tree storage"

Check out the uid and gid of the mounted directory /mnt/bbb:
```
ls -al /mnt/bbb/
```

Add an entry on /etc/fstab for mounting the storagebox directory to /mnt/bbb:

STORAGEBOX_HOST=uXXXXXX@uXXXXXX.your-storagebox.de
STORAGEBOX_DIR=storage/bbb.example.org/data
LOCAL_MOUNT_POINT=/mnt/bbb
STORAGEBOX_SSH_KEY=/root/storagebox/key1
MAP_UID=1000997
MAP_GID=1000996
SSHFS_OPTIONS=x-systemd.automount,x-systemd.requires=network-online.target,_netdev,user,idmap=user,transform_symlinks,port=23,identityfile=$STORAGEBOX_SSH_KEY,allow_other,default_permissions,uid=$MAP_UID,gid=$MAP_GID

cat <<EOF >> /etc/fstab
$STORAGEBOX_HOST:$STORAGEBOX_DIR  $LOCAL_MOUNT_POINT  fuse.sshfs  $SSHFS_OPTIONS   0  0
EOF
cat /etc/fstab

We are appending a single line, but if we break it for readability, it looks like this:

uXXXXXX@uXXXXXX.your-storagebox.de:storage/bbb.example.org/data \
    /mnt/bbb \
    fuse.sshfs \
    x-systemd.automount,\
        x-systemd.requires=network-online.target,\
        _netdev,\
        user,\
        idmap=user,\
        transform_symlinks,\
        port=23,\
        identityfile=/root/storagebox/key1,\
        allow_other,\
        default_permissions,\
        uid=1000997,\
        gid=1000996\
    0 \
    0

Mount the directory:

mount /mnt/bbb
systemctl daemon-reload
umount /mnt/bbb
mount /mnt/bbb

6.1.5. Restore the BBB data

incus restart bbb
incus shell bbb
bbb-conf --stop

rsync -a /var/bigbluebutton-1/ /var/bigbluebutton
chown bigbluebutton: -R /var/bigbluebutton
rm -rf /var/bigbluebutton-1/

bbb-conf --start
exit

6.2. Backup

If not using an external storage for the data (as described in the previous section), it might be useful to store there a copy of the data (as a backup).

See the section Access the storagebox with SSH for details on how to setup access via ssh keys to a StorageBox server.

This script, that is executed on the host, will mirror the directories /var/bigbluebutton/ and /root/ of BBB to the server storagebox:

cat << '_EOF_' > backup-bbb.sh
#!/bin/bash -x

CONTAINER=bbb
rsync="rsync -arAX --delete --links --stats -h"
destination="storagebox:backups/$CONTAINER"

main() {
    incus stop $CONTAINER
    mount_root_of_container

    # rsync /root
    $rsync mnt/root/ $destination/root/

    # rsync /var/bigbluebutton
    $rsync mnt/var/bigbluebutton/ $destination/var/bigbluebutton/

    unmount_root_of_container
    incus start $CONTAINER

    show_disk_usage
}

mount_root_of_container() {
    mkdir -p mnt
    incus file mount $CONTAINER/ mnt/ &
    pid=$!
    sleep 2
}

unmount_root_of_container() {
    kill -9 $pid
    sleep 2
    rmdir mnt
}

show_disk_usage() {
    echo "========== disk usage ==========="
    ssh storagebox du -hs 'backups/*'
    ssh storagebox df -h
}

# call main
main "$@"
_EOF_

chmod +x backup-bbb.sh
./backup-bbb.sh