TSD Operational Log - Page 5

The matlab license on Linux hosts expired. We are working to solve the problem as quickly as possible.

Consent and Publication internal portals are not available. Investigation is in progress ¡­

The Slurm controller on Colossus was unresponsive between approximately 01:00 and 09:15 on February 17th.

The issue has been resolved, but we advice users that had running jobs during this time frame to check their job logs for issues.

Resolved

We will be performing an upgrade of the Horizon view infrastructure on Thursday February 3rd at 12:00. We expect this to take no more than 2 hours. We will update this message and send out an email once the upgrade is complete. 

Login through view.tsd.usit.no will be down during the downtime. The project machines will not be affected, so any existing sessions you have will still be there when you log back on after the downtime.

Due to a software error, the Windows access of some users was extended past the period to which the project Administrator has approved. Now this is corrected. If you have lost your Windows machine access, please contact your project administrator to add you to the appropriate group. 

f4transkript is currently not available in TSD-projects.

We are working to solve the problem as quickly as possible.
Our apologies for the inconvenience.

We will be performing network maintenance on Monday 13/12/21 at 12:00, the downtime will last 10-15mins.

We do not expect any downtime, but there might be some interruptions. 

Between approximately 10:00 and 10:30, the module load command did not work on the submit hosts and Colossus nodes. Please resubmit any jobs that may have failed. This has been resolved.

We will be performing maintenance on view-ous.tsd.usit.no on Tue 16/11 12:00.

The downtime will last about an hour.

BankID is having planned maintenance downtime on Thursday 2021-10-21 00:01¨C08:01, and will not be usable for authentication while this work is ongoing.

For TSD, this means that login to TSD Selfservice via BankID and authentication to Nettskjema forms using BankID will be unavailable in the above specified timeframe.

For further information please see Digdir's downtime notice:
https://status.digdir.no/incidents/pnlr6gn69dy4

-----

Update: BankID for mobile is working again. BankID with ¡°kodebrikke¡± is still unavailable. This will also be unavailable on Oct 23

Update 20:30: Maintenance on the ESS storage system has been completed and the submit hosts are available again. Some submit hosts have been rebooted.

Update 18:58: Computation on Colossus has been resumed.

Update 18:48: The downtime has been extended, since the final update will result in additional NFS hangs.

Update 16:20: The downtime will be extended by approximately 2 hours. There may be additional NFS hangs on hosts mounting /cluster during this period.

Update 16:00: The last few Colossus nodes are being upgraded. Colossus will resume after that.

Colossus will have downtime on Wednesday 13/10 due to third-party maintenance on the /cluster file system. The affects /cluster on Colossus as well as NFS mounts of /cluster on the submit hosts.

Submit hosts will experience intermittent NFS hangs across the day.

Any jobs submitted to Colossus that cannot completed before the scheduled downtime w...

Hosts mounting /cluster/projects/pXX and /cluster/software experienced several NFS hangs on Saturday 09/10 and Sunday 10/10. User processes accessing these shares were killed upon remounting the NFS mounts.

The issue has been resolved.

We will be performing maintennance on the horizon view login servers, the downtime is estimated to last 1 hour. 

https://view.tsd.usit.no and https://view-ous.tsd.usit.no will be unavailable during the downtime.

--

The maintennance is now complete.

Nettskjema will no longer automatically deliver encrypted JSON-files to all TSD-projects.

This will not affect the use of Dataloader.

If you still want to recieve encrypted JSON files for each submission, please notify tsd-drift@usit.uio.no

Colossus will have downtime on Wednesday 13/10 due to third-party maintenance on the /cluster file system. The affects /cluster on Colossus as well as NFS mounts of /cluster on the submit hosts.

Any jobs submitted to Colossus that cannot completed before the scheduled downtime will remain pending until after the downtime. They will be rescheduled automatically once the downtime ends.

During the downtime we advice you to keep an eye on our operational log for any updates.

VMware View is down. We are working on it.

Best

TSD

Login to TSD is currently unavailable.
We are working to solve the problem as quickly as possible.
Our apologies for the inconvenience.

-- 
The TSD Team

We are doing network maintenance on Monday 30/8.

We do not expect any downtime during the maintenance.

We're experiencing NFS hangs on data/durable and cluster which started at 14:15-15:00. We're actively working on a solution.

A few hosts were rebooted.

Dragen has been updated to version 3.8.4 and new licenses have been installed.

The Slurm queue system was unresponsive between approximately 01:00 and 09:30 on 2021-06-30, indicated by this Slurm error: "slurm_load_jobs error: Socket timed out on send/recv operation"

No jobs have started in that period, but running jobs should not have been affected. If you had running jobs in this period we advice you to check your job results for errors.

The issue has been resolved.