TSD Operational Log - Page 12

Update to web-based file uploads. After this change files uploaded with https://data.tsd.usit.no and the tsd-api-client will be located in /data/durable/file-import/pXX-member-group, instead of the previous location: /data/durable/file-api.

The SPSS license is currently not valid. We are trying to update it as soon as possible. Thanks for your patience.

TSD

Dear TSD-users,

Unfortunately, our services are currently unavailable due to a DNS-issue. We're aware of the problem and working on solving this as quickly as possible.

Our apologies for the inconvenience.

Best regards,
The TSD-team.

There are currently problems with the 2 factor authentication, which makes new logins to TSD impossible. (Existing connections are not affected.)

A side effect is also that syncronization of new QR code keys has stopped.

Update: The reason for the downtime was a failed synchronization. Everything should be working now, including newly generated QR codes. If you are still experiencing problems, please contact us.

We will perform a scheduled system upgrade of Colossus starting at 2019-01-03, 10:00 until 2019-01-04, 10:00.

UPDATE (2019-01-04, 12:35)

The upgrade of Colossus is complete.

UPDATE (2019-01-04, 09:45)

We are experiencing a slight delay, and hopefully Colossus will be available for use by 13:00 CET.

TSD@USIT

Lately, the usage of Colossus has increased, and most of the time every node is busy running the jobs.  The Infiniband network on one of the storage nodes has been unstable lately, causing decreased performance and lower availability. We will attempt to correct this on Friday between 09:00 - 11:00 CET.

We will place a reservation on the queue system during the aforementioned timeframe. This way the new jobs that overlap the timeframe will not start, while allowing currently running jobs to finish.

Submitted jobs that are not specified to finish before Friday at 09:00 will remain queued until the maintenance is finished.

We are sorry for the inconvenience.

TSD@USIT

TSD is inaccessible for the moment, and all services are affected. We are working to correct the issue.

TSD@USIT

Some of Our users are unable to login to TSD through ThinLinc. We are working on resolving this issue.

TSD@USIT

Dear TSD users,

Due to a network issue, the service is currently unavailable. We're working on solving this as quickly as possible, and will update this message as we progress.

Our apologies for the inconvenience.

-- 
Best regards,
The TSD team

Login to TSD through view-ous.tsd.usit.no is not working for the moment, and we are working on resolving this issue.

TSD@USIT

We are experiencing problems with the FileLock.  Until the issue is solved user are advised to use the new Web File uploader for imports: /english/services/it/research/sensitive-data/use-tsd/import-export/index.html#toc4

The planned maintenance work in TSD has started. The TSD services will not be available today between 13:00 - 16:00 CET.

(Update: 15:56): All services are working.

Kind Regards,

TSD@USIT

Due to storage capacity problems, jobs have been pending on Colossus. We will update this status when we have resolved this issue.

Several issues makes most services in TSD unavailable. We are investigating the problems, and will come back with an update.

Update (12:05): Because one of the virtualisation clusters crashed, many VMs were forcefully restarted. Most services is back up now, but some services will need manual interactions.

Update (12:30): All services should behave normally now.

We are experiencing issues with some services, which may lead to some users being unable to login to TSD. We are investigating the cause of this and working on fix.

Solved: Some services hanged after a unplanned reboot of a part of our infrastructure last night. We have restarted the services that seemed to be affected, and all services should be up now.

On first of October between 13:00 - 16:00 CET, our team of engineers will perform an infrastructure upgrade. This upgrade is necessary, as we need more VLANs for our increasing growth of projects in TSD.

The downtime will affect all our services, so please do not schedule any long running jobs during this time. Please, also save your data before the maintenance window, and follow our Operation Log for the update:
http://www.uio.no/english/services/it/research/sensitive-data/log/

We are sorry for the inconvenience.

Kind Regards,
TSD@USIT
 

Dear TSD User,

There will be a short maintenance window on Wednesday 22nd of August.

From 12:00 until 12:30 CET, there will be interruptions to the Connections with the network file system, and the shared folders will not be accessible for a short period. Please save your data before the maintenance window and follow our Operation Log for the update.

As soon as our engineers are done with the maintenance, we will update this Operational Log.

We are sorry for the inconvenience.

Kind Regards,

TSD@USIT

Dear TSD-User,

We will be performing a short scheduled maintenance of our Connection servers in TSD. Between 13:00 - 13:05 you will not be able to login to Windows machines. Users that are already logged in will not be affected by this downtime.

Dear TSD-users,

At the moment, Windows virtual machines in TSD are unable to mount \\tsd-evs. We are working in tandem with the storage team to resolve this issue as quickly as possible.

Our apologies for the inconvenience.

-- 
Best regards,
The TSD team

TSD is inaccessible for the moment. The engineering team is actively working to correct the issue.

Due to the planned power outage today, we had problems with the NFS-export, and therefore some jobs on Colossus were affected by this. The problem is now fixed, and we are sorry for the inconvenience.

TSD@USIT

Dear TSD users!

25.06.2018 @ 13:00 - 14:00 we will be performing a short scheduled maintenance of the central filesystem in TSD. Some of you might experience short hiccups in accessing your files during this time frame.

-- 
Best regards,
TSD team

Dear TSD users,

We regret to inform you, that due to sudden increase in concurrent Linux logins over the last two weeks, we've reached our limit for concurrent logins.

This means that some of you might not be able to log in to your Linux-VMs using the conventional method of https://login.tl.tsd.usit.no.

We have ordered more licenses and are eagerly waiting for them to arrive.

In the meantime:

You can log in to your Linux-VMs by first logging in to your Windows-VM using the following address: https://view.tsd.usit.no and then opening the application Putty to SSH to your vm. The hostname for your machine will have the following format, here using project p11 as an example: p11-tl01-l.tsd.usit.no

-- 
Best regards,
TSD

Some projects could not login to their windows machine through view. The incident took place during the time slot:

Start:          2018-05-13 13:00
End:            2018-05-14 10:00

We apologize for the inconvenience

TSD@USIT