TSD Operational Log - Page 16

Due to a jump host failure, TSD was not accessible 09:15 - 14:31 today. The failure has also caused NFS hangs on some Linux VMs, due to which we had to reboot those VMs. All thinlinc VMs now should be accessible. Some other Linux VMs my still have problems. We are working in fixing those ASAP

TSD@USIT

Due to NFS hangs, some Linux hosts are now inaccessible. The issue may result in TSD being inaccessible via thinlinc. This should not affect log in through VMWare Horizon.

We doing our best for this issue to be resolved ASAP

TSD@USIT

---------------------------------------------------

The issue was resolved after the affected hosts have been rebooted

There will be a maintenance stop on the 02/05 at 13:00 CET for one hour. During the downtime, the linux and windows VMs will not be accessible. The Colossus cluster will be under maintenance.
 

We are having an issue with the main gateway and the login to TSD in this moment is not possible. We are working to solve the problem as soon as possible.

We apologize for the inconvenience.

----

Update 14/04, 12:15 -- TSD partially operational. Still not possible to submit jobs to Colossus

----

Update 18/04, 09:40 -- Still there might be issues when submitting jobs to  Colossus

The problem was solved on Friday around 22:00. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

----

TSD is unreachable at this moment. We are trying to solve the problem as soon as possible. We apologize for the inconvenience.

The TSD gateway was not reachable between 15:43 and 16:15 today, the issue has been resolved and users should be able to log in again.

Some linux machines might still not be available, if you are unable to log into ThinLinc please contact us.

The problem was solved the same day around 18:10. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

----------------

TSD's main gateway is not reachable at the moment and the TSD infrastructure is not accessible. We are trying to solve the situation as soon as possible. More information will appear on the operational log on Monday 27/02 in the morning.

We apologize for the inconvenience.

The problem was solved on Friday around 17:15. It might be that some of the linux VM are not yet reachable. In that case please mail us and we will restart the machine.

------------------

TSD is unreachable at this moment. We are trying to solve the problem as soon as possible. We apologize for the inconvenience.

We have had a failure of the primary jumphost  but the failover machanism took over and moved the system to the secondary one. The infrastructure shall be back to normal very soon. We are investigating what caused the failure.

We apologize for the inconvenience.

We are having some network problem at the moment and the services in TSD is not reachable. We are working to investigate the causes in order to solve the problem as soon as possible.

We apologize for the inconvenience.

Regards

Nihal@TSD

Dear TSD-users!

There is an issue accessing the colossus storage at the moment. This may result in /cluster/projects/pXX being inaccessible.

We are working on this issue to get it resolved ASAP.

We apologize for the inconvenience.

Regards,

Nihal @TSD

Dear TSD-users!

Due to a disk failure on tsd-fx01.tsd.usit.no which happened 2017-01-20, 22:59, the file import in TSD has been unavailable until 2017-01-23, 10:56.

The issue has been resolved, and the system is back in production.

We apologize for the inconvenience.

Regards,
Benjamin

There is an issue accessing the colossus storage at the moment. This may result in:

• /cluster/projects/pXX being inaccessible
• Software modules being inaccessible
• Issues submitting to Slurm

We are working on this issue to get it resolved ASAP.

We apologize for the inconvenience.

Regards,

Abdulrahman

Due the network problem we had yesterday morning, the Colossus disk was not properly mounted and exported to the project machines. This results in problem accessing the /cluster/projects partition, mounting modulefiles and running slurm. Some of the jobs that were running when the network issue occurred might have been affected by problem.

We are on the issue now, hoping to solve it as soon as possible during the day.

We apologize for the inconvenience.

Francesca

Update: Problem is fixed. You can now login into TSD.

---------------------------------------------------------------------------------------------------

We are having some network problem at the moment and the TSD is not reachable. We are working to investigate the causes in order to solve the problem as soon as possible.

We apologize for the inconvenience.

Regards,

Erik

Due to a failure in the heating system, the Colossus front-end and one of the rack went down this morning (05/12-2016) and the cluster is not available at the moment. We are working to reboot the system at the moment.

Jobs that were running on the rack that went down, unavoidably died while those running on the other rack are most likely running even though the front end is not available.

We apologize for the inconvenience.

Dear TSD users,

There has been a network problem, which has been safely bypassed by our failover mechanism. However some of the linux VMs are still mounting the filesystem and therefore  are not accessible at the moment. The process will take 2 hours. If after 11:00 today you still experience problem with your linux VM, please let us know (tsd-drift@usit.uio.no).

We are investigating the cause of the network problem.

We apologize for the inconvenience.

Regards,

Francesca

Dear TSD-users,

The service is back to normal now.

We apologize for the inconvenience,

Regards

Nihal @TSD

Dear TSD-users,

we are experiencing problem on the Filelock filesystem. The service is in function but extremely slow. We are investigating the causes, hoping to solve the problem as soon as possible.

We apologize for the inconvenience,

Regards,

Francesca@TSD

Dear TSD-users,

we need to reboot the machine that allows to two factor authentication. The reboot will happen at 13:30 today and  it will take around 5 minutes. During the reboot it will not be possible to perform any login to TSD. Sessions already opened will remain opened.

We apologize for the inconvenience,

Regards,

Francesca@TSD

Dear TSD-linux users,

it has been found a security VULNERABILITY on the linux RHEL6 kernel and the linux machines (physical and virtual) in TSD have been rebooted during the night. This was absolutely needed. We are now working to reset those virtual linux client that hasn¡¯t rebooted properly. It will take up to one hour.

Please follow the operations on our operational log.

We apologise for the inconvenience.
Regards,
Francesca@TSD

Dear TSD-user

We encountered a bug on our file server a couple of weeks ago, and to resolve this issue, quota statistics on the HNAS disk usage had to be reset. It will take up to two weeks until you will be able to see the correct quota statistics again. In the meantime you have the disk available but there is no management engine to enforce quotas on it. To avoid severe incident, we kindly request you, for the next three weeks, to inform us in advance if you need extra disk space (more then 1TiB) and we will try to adjust your request. Please do not import or produce amount of date for more then 1TiB without informing us.

Please notice that this message regards only the usage of HNAS disk, and not the usage of Colossus disk.

We apologize for any inconvenience this may cause you.

Regards,

Francesca@TSD

Dear TSD-users,

the maintenance is finished according to plan today at 25/10 at 16:00. The service is back in production.

Regards,

Francesca