Backup/Restore/Copy Course via the Web Interface
One of the more common Moodle-related admin tasks is creating new courses at the start of each semester.
Sometimes instructors will request blank courses. To do this:
- Go to Admin -> Courses -> Manage Courses & Categories,
- Navigate to the category for the current semester
- Select create course and fill in the information (title, short name, start date).
- The start date should be the starting date of the semester since all dates in the course are keyed off of that start.
- The course short name should be of the form CSCInnnn-semYY where ‘sem’ is F, S or Su and YY is the year digits.
- The course title should be of the form CSCInnnn - faculty - title.
- When the course is created, navigate to the course, and add the instructor as ‘teacher’. They can then add other teachers (TA’s, etc).
More often than not, faculty want a copy of a previous semester’s course. To create such a copy, one must first backup the course using Moodle built-in course backup features, and then restore the course to the new semester. You can either use a course from the S3 archive or from prior semesters. To restore from a prior semester that is still on Moodle:
- Navigate to the class instance
- Go to Backup in the course admin menu
- Backup the course
- Select the “restore” option.
You would restore the course as a new course in the new semester category. When you restore the course, insure that you’re not also restoring the users. Once restored, modify the title and the short name to match the appropriate faculty and semester.
Initial Command Line Environment Setup
gcloud auth application-default login
gcloud config set project emerald-agility-749
gcloud container clusters list
gcloud container clusters get-credentials production --zone us-west1-a
Restore from backup after system failure
Google Compute is configured to make nightly snapshots of the SQL database
and Moodle data disk going back 60 days. You can view those snapshots in the project level
dashboard (Compute Engine -> Snapshots).
In the event of total failure. Here are the steps to restore the data.
- Bring down all k8s cluster. You cannot restore disks that are mounted.
- Create a new disk.
- Specify name (note, you’ll need to delete the old disk before, so do this carefully. Maybe test first with a different name)
- Build the disk from snapshot and select what date you want to backup from
- Run Create.
To restore the SQL data, the steps are even easier.
- Go to SQL dashboard in GCE.
- Go to backups
- Right click on backup and click restore and select target instance. Note, you will need to remove the High Availability before GCE will let you do this.
- Backup
Once the data has been restored, you can redeploy the k8s environment.
It is essential to archive all Moodle course content at the end of each semester. Moodle is used for ABET accreditation and the full backups (including user information) are necessary for that.
Current backups are archived to Amazon Glacier using the departmental AWS account. Backups can be done by logging into Moodle, using moodle/admin/cli/backup.php to produce MBZ files and then transferring those to the S3 bucket labeled “moodle-backups”. Archives are stored by year and semester.
The backup.php requires knowing the course id number for the course. These can be determined using the Moosh utility (see moosh.org) and the course-list command. Building complete backups takes a while (couple of hours)
For more details, the Moodle backup and restore docs can be found at:
- https://docs.moodle.org/en/Backup
Moosh Install
- Download the version of Moosh that matches the version of Moodle in use- Moosh Utilities
- Extract the ZIP file
unzip moosh_moodleVV_YYYYMMDDxy.zip
- Tar the output of the zip file (Containers do not have unzip support)
tar -cvzf moosh.tgz moosh
- Find a php-fpm pod to work with
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php-fpm
- Copy the tar file to a php-fpm pod
kubectl cp moosh.tgz php-fpm-XXXXXXXX-XXXX:/srv/moodle/
- Connect to the containter
kubectl exec -it php-fpm-XXXXXXXX-XXXX /bin/bash
- Extract Moosh
cd /srv/moodle gunzip moosh.tgz tar -xvf moosh.tar rm moosh.tar
Manual Course Backup & Restore
There are large courses in Moodle that cannot be completed using the web interface. Attempting to Backup one of these courses will result in the Backup Complete page not being shown. Instead you will be presented with a blank content page or a progress bar that does not progress. When this occurs, you will need to use the Moosh at the command line to process the backup and restore.
- Login to a php-fpm pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php-fpm kubectl exec -it php-fpm-XXXXXXXX-XXXX /bin/bash
- Backup the course by COURSE_ID
cd /srv/moodle/code su -s /bin/bash -c "/srv/moodle/moosh/moosh.php course-backup -f /tmp/[COURSE_ID].mbz [COURSE_ID]" www-data
- Determine the Category ID where the course will be restored
su -s /bin/bash -c "/srv/moodle/moosh/moosh.php category-list" www-data
- Restore the course
su -s /bin/bash -c "/srv/moodle/moosh/moosh.php course-restore /tmp/[COURSE_ID].mbz [CATEGORY_ID]" www-data
- Clean up
rm /tmp/[COURSE_ID].mbz
- Open the Moodle web page and Login
- Navigate to the new course
- Fix the Name, Short Name, Start Date
- Navigate to the course Participants
- Select all Participants and delete them
- Enroll the new teacher as a Teacher in the class
- Send an email to the teacher with a link to the course page
Problem: Email not sending
From time to time, Moodle is unable to send out the regular email notifications. To confirm the problem:
- Navigate to Moodle -> Site Administration -> Server -> Email Test
- Provide a “To email address”
- Click “Send a test message”
If the next page is not Success, follow the steps below to restart the SMTP pod.
- Identify the SMTP pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep smtp
- Kill the existing pod
kubectl delete pod [SMTP pod name]
- Wait for the new pod to reach Running state
kubectl get pod | grep smtp
- Resend the test email as described above to confirm it is working again
Deploy Moodle to DEV
This build process is dependant on the following utilities being installed
- gcloud kubectl (this part of gcloud components)
- envsubst (brew install gettext)
- realpath (brew tap iveney/mocha && brew install realpath)
- Connect to Development
gcloud config set project csel-dev-161517 gcloud config list kubectl config use-context gke_csel-dev-161517_us-west1-a_development kubectl config set-context gke_csel-dev-161517_us-west1-a_development --namespace development
- Restore the SQL development database with production data in the Cloud Console.
- Click SQL from the left side navigation
- Click moodle-mysql-dev from the content area
- Click the Start button to start the dev instance
- Click Back to return to the SQL server list
- Click moodle-mysql from the content area
- Click the Backups tab
- Find the most recent backup and click on the 3-dot button to its right
- From the drop-down menu, click Restore
- Set the Target Instance drop-down to “moodle-mysql-dev”
- Click OK
- Restore the Moodle data disk.
- This process is not ideal but Google Compute does not allow you to access snapshot disks across projects, however, it does allow you to share images. So we first create an image from a snapshot and share this with the development project.
- Create an Image of the most recent moodle-data-disk-ssd
- Click the project name at the top of the page
- In the Select From drop-down, select No organization
- Click CSEL from the project list
- Click Compute Engine from the left side navigation
- Click Images from the left side navigation
- Create Create Image from the top tool bar
- Name the image moodle-data-disk-ssd-image-YYYYMMDD
- From the Source drop-down, select Snapshot
- In the Source snapshot drop-down, select the gcs-moodle-data-disk-ssd-moodle-data-XXXXXX with the largest number
- Click the Create button at the bottom of the form
- Wait for the image to be created
- Find the selflink path for the new image
- Click Compute Engine from the left side navigation
- Click Images from the left side navigation
- Click on the name of the image from step 1-4
- Click Equivalent REST at the bottom of the page
- Find the “selfLink” value and copy it for use in creating the new disk
- Delete the existing moodle-data-disk-ssd in Development
- Click the project name at the top of the page
- In the Select From drop-down, select colorado.edu
- Click CSEL-DEV from the project list
- From the left navigation, Click Disks (under Compute Engine)
- Double check that you are in the CSEL-DEV project
- Find the moodle-data-disk-ssd
- Click the 3-dot button to the right
- Click Delete from the drop-down menu
- Triple check that you are in the CSEL-DEV project
- Click the Delete button
- Create a new moodle-data-disk-ssd in development.
- Run the same window you preformed step 1, run the following command after replacing [SELFLINK_PATH] with the selflink path captured when creating the new image
gcloud compute disks create moodle-data-disk-ssd --zone=us-west1-a --image=[SELFLINK_PATH]
-
Determine the stable version of Moodle for the upgrade https://download.moodle.org/releases/latest/
- Clone the Moodle repo (or pull updates if you already have it)
git clone git@bitbucket.org:ucbcsops/k8s-moodle.git
- Clone the Registry (or pull update if you already have it)
git clone git@bitbucket.org:ucbcsops/registry.git
- Edit the develop variables file
cd k8s-moodle vi environments/development/variables.conf
- Change the JENKINS_BUILD variable to match the desired version number and set the build number
Note: Change version number to match what you found in step 3 and the revision number (-1) should match the build number in Jenkins.
export JENKINS_BUILD=development3.6.3-1
- Push the change to bitbucket
git add environments/development/variables.conf git commit -m "Update Jenkins build number" git push cd ..
- Create a development branch of the registry
Note: Change version number to match
cd registry git checkout -b development3.6.3
- Edit the Moodle-base Dockerfile
vi moodle-base/Dockerfile
- Change the Moodle version to match the desired version number
Note: Change version number to match what you found in step #3
ENV MOODLE_VERSION v3.6.3
-
Download the matching (or most recent) REMUI theme (as needed) to registry/moodle-ucbcs/theme EdWiser
- Change into the theme directory
cd moodle-ucbcs/theme
- Unzip the new Theme
unzip Edwiser_Remui_3_443e8cd7b289e3d44.zip
- Unzip each of the nested Zip files
unzip Edwiser_Course_Formats_v1.0.0.zip unzip edwiser_remui_3.6.2.zip unzip edwiser_remui_block_1.0.3.zip
- Move each into a versioned folder
Note: REMUI 3.6.2 was the latest version when this was written
mv remui remui_3.6.2 mv remuiblck remuiblck_3.6.2 mv remuiformat remuiformat_3.6.2
- Cleanup zip files
rm *.zip
- Force hiding of the local login option
vi remui_3.6.2/classes/output/core_renderer.php
- Find the login box text
// sign in popup
- Add the style the form tag
style="display: none"
- Edit the Moodle-ucbcs header fragment
vi ../fragments/0-header.fragment
- Modify the theme paths
ADD theme/remui_3.6.2 ${MOODLE_CODE_DIR}/theme/remui ADD theme/remuiblck_3.6.2 ${MOODLE_CODE_DIR}/blocks/remuiblck ADD theme/remuiformat_3.6.2 ${MOODLE_CODE_DIR}/course/format/remuiformat
- Push branch to repo
cd ../.. git add . git commit -m "Moodle DEV upgrade" git push --set-upstream origin development3.6.2
- Change into the k8s-moodle directory
cd ../../../k8s-moodle
- Build the environment
./build-env development
- Verify environment data
> [1] stage: load environment data > - checking for environment folder at environments/development... > - variables file found, attempting to load it... > - all variables successfully loaded: > > - environment hostname: moodle-dev.csel.io > - container registry : gcr.io/csel-dev-161517 > - load balancer addr : 35.185.192.128 > - k8s target namespace: development`
- Verify NFS IP is sane
> [4] stage: create NFS service to obtain IP > - templating the NFS service configuration file only... > - single file created at build/development/workloads/nfs/svc.yaml > - creating the NFS server service... > - kubectl: service "nfs-server" created > - attempting to obtain the NFS service IP address... > - service address: 10.43.250.93
- Fix permissions on scripts
chmod 750 build/development/namespace/* chmod 750 build/development/scripts/*
-
Check Jenkins for build completion (Must be on CU VPN)
- Tear down all services in DEV
build/development/scripts/tear-down-all
Note: a remaining cluster node is okay
- Ensure all pods are gone
kubectl get pod
- Deploy the environment
build/development/scripts/deploy-all
Error messages about NFS already existing are okay/expected.
- Test (Must be on CU VPN) Moodle-dev
Remember: When you are done Testing, stop all services and shutdown the moodle-mysql-dev database, test cluster, and test disks
When you are ready to update production
kubectl edit deployment php-fpm
Update to match the new version you tested in DEV
Additional Documentation Link(s)
https://bitbucket.org/ucbcsops/k8s-moodle/src/master/
Moodle DEV testing checklist
- QA Test
- NFS Data Mount
- Run Benchmark
- Code Runner
- Course Content (Y)
- Qngine
- Quiz Content
- Shib Login/Logout
- Manual Login/Logout
- Course Category
- Course Page
- Grades
- Participants
- Enroll Users
- Dashboard
- LTI
- STACK
- H5P
- Turn Editing On (Teacher)
- Calendar
- Change Roles
- Add Resource in Course
- Site Admin Console
- Memchached (Clear)
- PHP Config Lib
- Create Test Course
- Backup Course
- Restore Course
Backup Moodle courses
- Login to Amazon S3
- Amazon Login
- Account: bouldercompsci
- Navigate to S3
-
Click the Create Folder Button
-
Privide a new folder name (YYYY-semester)
-
Click Save
- Connect to a Moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php kubectl exec -it [PHP POD NAME] /bin/bash
- Move into the Moodle root
cd /srv/moodle/code
- Change into the Moodle user
su -s /bin/bash www-data
- Create backup location
mkdir /tmp/backups
- Identify the category ID to be backed-up
/srv/moodle/moosh/moosh.php category-list
- Add the backup script (replace [CATEGORY_ID] with the desired number from step 10)
cat > /tmp/backup.sh << EOF for COURSEID in \$( /srv/moodle/moosh/moosh.php course-list -c [CATEGORY_ID] -f id | grep \" | tail -n +2 | tr -d \" ); do echo Processing \$COURSEID; /usr/bin/php /srv/moodle/code/admin/cli/backup.php --courseid=\$COURSEID --destination=/tmp/backups; rm -rf /srv/moodle/data/temp/backup/* ; rm -rf /srv/moodle/data/trashdir ; done EOF
- Start the backup
chmod 700 /tmp/backup.sh nohup /tmp/backup.sh > /tmp/nohup.txt &
- Monitor progress until completion
tail -f /tmp/nohup.txt
- Copy backups to Amazon S3
- Remove local backups
rm -rf /tmp/backups rm /tmp/backup.sh
Install S3 PHP v3
-
Go to AWS S3 PHP installation instructions AWS S3 PHP Installation Instructions
-
Download the ZIP file from under the Installing by Using the ZIP file header
-
Open a terminal window
- Copy the ZIP file into a new folder
cd /tmp mkdir awsphp cd awsphp mv aws.zip /tmp/awsphp
- Extract the ZIP
cd awsphp unzip aws.zip
- Remove the ZIP
rm aws.zip
- Recompress ZIP as Gzipped TAR from the parent directory
cd .. tar -cvzf /tmp/awsphp.tgz awsphp
- Copy file to Moodle PHP
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php kubectl cp awsphp.tgz [PHP POD NAME]:/tmp/
- Login to a Moodle PHP container
kubectl exec -it [PHP POD NAME] /bin/bash
- Decompress archive and remove
cd /tmp gzip -dc awsphp.tgz | tar -xvf - rm awsphp.tgz
PHP S3 Upload script
Prerequisites: Install S3 PHP v3
- Connect to a Moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php kubectl exec -it [PHP POD NAME] /bin/bash
- Create backup script (Key and secret must be replaced. See the vault for actual values.)
cat > /tmp/s3backup.php << EOF #!/usr/bin/env php <?php require '/tmp/awsphp/aws-autoloader.php'; use Aws\Common\Exception\MultipartUploadException; use Aws\S3\MultipartUploader; use Aws\S3\S3Client; // f = filename (in current directory) // t = Term (e.g. 2019-spring, 2019-summer) folder name \$arguments = getopt('f:t:'); \$bucket = 'cu-cs-moodle-backups'; \$keyname = \$arguments['t'] . '/' . \$arguments['f']; \$s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-west-2', 'credentials' => [ 'key' => 'ACCESS_KEY_ID', 'secret' => 'SECRET_ACCESS_KEY' ] ]); // Prepare the upload parameters. \$uploader = new MultipartUploader(\$s3, \$arguments['f'], [ 'bucket' => \$bucket, 'key' => \$keyname, 'ACL' => 'private' ]); // Perform the upload. try { \$result = \$uploader->upload(); echo "Upload complete: {\$result['ObjectURL']}" . PHP_EOL; } catch (MultipartUploadException \$e) { echo \$e->getMessage() . PHP_EOL; } ?> EOF
- Run backup script for each backup
/tmp/s3backup.php -t [TERM] -f [BACKUP_FILE_NAME]
Term from Backup Moodle courses step #4
- Login to S3
- Navigate to the term’s upload location
- The number of files is listed at the right side of the header and footer of the file list table.
- Confirm all backups are present
- Login to Moodle
- Click Site adminstration
- Click the Courses tab
- Click Manage courses and categories
- Under the Course categories see the number at the end of the category entry and compare to the number of uploaded files in the S3 bucket’s term folder.
- Change Storage Class on all backups to Glacier
- Checkbox the term folder
- From the Action drop-down click Change storage class
- Click the Glacier radio-button
- Click Save
- Delete the backups
rm -rf /tmp/backups
- Delete supporting script
rm /tmp/s3backup.php rm /tmp/backup.sh
PHP S3 Download script
Prerequisites: Install S3 PHP v3
- Connect to a Moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php kubectl exec -it [PHP POD NAME] /bin/bash
- Create backup script (Key and secret must be replaced. See the vault for actual values.)
cat > /tmp/s3download.php << EOF #!/usr/bin/env php <?php require '/tmp/awsphp/aws-autoloader.php'; use Aws\S3\S3Client; use Aws\Exception\AwsException; \$arguments = getopt('f:t:'); if (count(\$arguments) != 2) { echo "\n\nGetObject.php -t <yyyy-term> -f <filename>\n\n"; exit(); } \$bucket = 'cu-cs-moodle-backups'; \$keyname = \$arguments['t'] . '/' . \$arguments['f']; try { \$s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-west-2', 'credentials' => [ 'key' => 'ACCESS_KEY_ID', 'secret' => 'SECRET_ACCESS_KEY' ] ]); \$result = \$s3->getObject(array( 'Bucket' => \$bucket, 'Key' => \$keyname, 'SaveAs' => \$arguments['f'] )); } catch (S3Exception \$e) { echo \$e->getMessage() . "\n"; } ?> EOF
- Make the script executable
chmod 700 /tmp/s3download.php
- Run backup script for each backup
/tmp/s3download.php -t [TERM] -f [BACKUP_FILE_NAME]
Delete a category with courses
WARNING: This will remove a category and all the nested courses. Ensure the courses have been backed up and use with extreme caution
- Connect to a moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep php kubectl exec -it [PHP POD NAME] /bin/bash
- Switch to the Moodle user
su -s /bin/bash www-data
- Move into the Moodle code base
cd /srv/moodle/code
- List existing categories
/srv/moodle/moosh/moosh.php category-list
- Delete the desired category (Replace [CATEGORY_ID] the desired category number from step 4)
/srv/moodle/moosh/moosh.php category-delete [CATEGORY_ID]
Expand Moodle data disk
- Using GCP console or gcloud CLI, increase the size of the moodle-data-disk-ssd
- Login to the NFS pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep nfs kubectl exec -it [PHP POD NAME] /bin/bash
- Find the moodle data disk device
df -h | grep exports
- Resize data disk (replace [DEVICE] with value from step 3)
resize2fs /dev/[DEVICE]
REMUI Update Nag Removal
Use this Custom CSS code in Custom CSS section inside Edwiser RemUI general settings-
.update-nag {
display:none
}
Manual SSL Certificate Refresh (Let’s Encrypt)
- Connect to a Moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep letsencrypt kubectl exec -it [POD NAME] /bin/bash
- Move into the script directory
cd /opt
- Execute the update script
./gen_le_cert.sh
- Reload the moodle webpage and confirm the new certificate is active
Alternatively, test it with a service like SSL Labs
Problem: Email sends, but is not delivered
There could be many possible problems, and here we will look at how to determine the root cause.
- Connect to a Moodle pod
gcloud config set project emerald-agility-749 kubectl config use-context gke_emerald-agility-749_us-west1-a_production kubectl config set-context gke_emerald-agility-749_us-west1-a_production --namespace moodle-prod kubectl get pod | grep smtp kubectl exec -it [POD NAME] /bin/sh
- Find the deferred mail spool
cd /var/spool/postfix/defer
- Select a queue
ls cd [AnyQueue] ls cat [AnyMessage]
-
Examine the message file contents for reason=
- Check external IP
cd /tmp wget https://ipchicken.com/ cat index.html | grep -A 1 Address rm index.html
This address should be 104.198.9.1 (if not assign that IP to the node Moodle node)
Alert! License is not activated , please activate the license in RemUI settings.
- Login to Moodle
- Navigate to Edwiser RemUI settings > License Status
- Click the Activate License button
If that does not resolve the problem do the following and contact EdWiser
- Navigate to Edwiser RemUI settings > General settings
- In the Custom CSS section add the following
.license-nag { display:none; }
- Click the Save changes button at the bottom of the page
Broken question (missing prototype ‘xxxxxxxxx’). Cannot be run
- Login to Moodle
- Navigate to the class and quiz generating the error message
- Click the Gear icon
- Click Question Bank
- From the Select a category drop-down, Select CR_PROTOTYPES
- Delete any duplicates (usually not edited by an admin)
- Repeat steps using LOCAL_PROTOTYPES form the drop-down