search -  faq -  rulez -  staff

forums






map cycle
irc              pwnalizer
voice         dswp bar




Register

Login


It is currently 04.28.25

 

All times are UTC + 1 hour




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Gameservers Down
PostPosted: 07.24.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4649
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1035.12
kills: 26022
deaths: 19948
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 810.059
kills: 3930
deaths: 3611
ratio: 1.08
hi everyone...
last days the game servers were down, its cause a disk died and there was no backup.
actually it was a mobile disk, usb, originally "temporarily" connected (and forgotten) over the last 10 years.
vmware doesnt even support usb drives natively, this setup was in fact left over from a migration.
hm.

the drive:
Attachment:
Screenshot from 2021-07-24 07-34-11.png
Screenshot from 2021-07-24 07-34-11.png [ 7.48 KiB | Viewed 7762 times ]


so i thought about 3 things.
- recovering the data from the disk and, ofc,
- getting a proper store and
- backup.

##### PART 1 RECOVERY #####

i dont have a car at the moment, the datacenter is some hours by train away.
blinky lives close there, so he went to datacenter to get that disk.
i myself bought some hardware and sent it to blinky before, so he could pimp the storage hw a bit.

result of blinkys inspection:
the disk comes up when connected.
gparted shows an unknown partition, but its vmware vmfs, thats normal:
Attachment:
Screenshot from 2021-07-24 07-43-46.png
Screenshot from 2021-07-24 07-43-46.png [ 599.56 KiB | Viewed 9480 times ]


but it fails then after some seconds. The warmer it gets, the faster it disconnects.
It disconnects usb-device-wise, so at least the drive seems ok.

now blinky had an idea.
this is the drive:
Attachment:
Screenshot from 2021-07-24 07-46-24.png
Screenshot from 2021-07-24 07-46-24.png [ 590.3 KiB | Viewed 9480 times ]
Attachment:
Screenshot from 2021-07-24 07-46-34.png
Screenshot from 2021-07-24 07-46-34.png [ 726.83 KiB | Viewed 9480 times ]


the board is a 2060-771801-002 REV A
it looks like swapping a board from an identical disk could bring the disk back.
but not the data. partitioning informations are stuck in the bios of the old hard disk.
fortunately, blinky got the soldering master degree, so we ordered a board...

this is todays state of the project, blinky swapped already the bios chips.
Attachment:
Screenshot from 2021-07-24 07-51-51.png
Screenshot from 2021-07-24 07-51-51.png [ 609.35 KiB | Viewed 9480 times ]


##### PART 2 NEW DISK #####

1.) Hardware...
a year ago, i started to build some storage.
basically, it was cause there was some time and money.
so i bought a dell 730xd as a first storage box...
Attachment:
Screenshot from 2021-07-24 08-11-38.png
Screenshot from 2021-07-24 08-11-38.png [ 785 KiB | Viewed 9480 times ]


and i got a pair of juniper ex4200 switches, so i could run ha storage and ip networks.
Attachment:
Screenshot from 2021-07-24 08-24-21.png
Screenshot from 2021-07-24 08-24-21.png [ 446.65 KiB | Viewed 9480 times ]


the 730xd is ideal imho, it has 25 sas slots and alot pcie slots/lanes for nvme.
the ex4200 have each 2x 10g uplinks, the dell server has 2x 10g.
so the switches have space for 2x storage with 2x 10g each.

anyhow, the hypervisors must be connected, too and theres jist 1g links left.
the ex4200 can aggregate 8 physical links, so thats what i planned:
every hv gets 8 links, there will be a dswitch with lacp in vmware.
additionally, the 730xd needs some cheap test disks...
here we are. intenso 240gb.
normally a bad choice but bigul checked them ok.
Attachment:
Screenshot from 2021-07-24 08-27-36.png
Screenshot from 2021-07-24 08-27-36.png [ 686.48 KiB | Viewed 9480 times ]

testing the packaging hardware for transport!
Attachment:
Screenshot from 2021-07-24 08-27-55.png
Screenshot from 2021-07-24 08-27-55.png [ 378.74 KiB | Viewed 7762 times ]


when blinky was in datacenter, he could mount nics and ssds.
so it looks like that now:
Attachment:
Screenshot from 2021-07-24 08-34-53.png
Screenshot from 2021-07-24 08-34-53.png [ 91.39 KiB | Viewed 7761 times ]

Attachment:
Screenshot from 2021-07-24 08-36-09.png
Screenshot from 2021-07-24 08-36-09.png [ 504.05 KiB | Viewed 9479 times ]


dswitch and lacp are on, next step will be playing with nfs4, vmware and multipathing...
Attachment:
Screenshot from 2021-07-24 08-38-50.png
Screenshot from 2021-07-24 08-38-50.png [ 119.39 KiB | Viewed 9479 times ]


expect an update sooner or later. (:
anyone interested in storage and ethernet?

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 07.24.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4649
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1035.12
kills: 26022
deaths: 19948
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 810.059
kills: 3930
deaths: 3611
ratio: 1.08
in vmware, theres several ways to connect storage.
for redundant connections, fibre channel ist widely known.
smaller installs may use iscsi or even old nfs.

now comes my ideas...
fibrechannel is no option, plus i think that block based storage belongs to local chassis.
vmware dont need block devices really.
the setup should combine storage and ip networking, saves alot space and power.

iscsi is the more modern way to go ip based.
but its block based, i wanted to get rid of this.

look how old nfs does true active/active multipathing:
Attachment:
Screenshot from 2021-07-24 11-29-25.png
Screenshot from 2021-07-24 11-29-25.png [ 37.15 KiB | Viewed 9471 times ]


the result is an extra 0,2 ms access time to the intenso lol ssd, as seen from a windows vm:
Attachment:
Screenshot from 2021-07-24 11-19-47.png
Screenshot from 2021-07-24 11-19-47.png [ 14.71 KiB | Viewed 9471 times ]

Attachment:
Screenshot from 2021-07-24 11-27-01.png
Screenshot from 2021-07-24 11-27-01.png [ 20.45 KiB | Viewed 7753 times ]


multipathing works through IP-hash-based link aggregation and nfs 4 does IP- based multipathing.
if you dont have a switch that can do link aggregation:
take 10 miniswitches, 10 network cards in each server, 10 subnets and you go.
you can even combine fast and slow links, which is one of many advantages of IP.

the other advantage is the file system.
in vmware, you cant choose a filesystem, theres only "vmfs"
recovering this failed disk means mounting it in vmware.
now i can have ext, xfs, i have even ram for zfs in the 730xd...

for the first i will track that read bandwith thingy, looks a bit mad with 200MB/s.
30K IOPS @ 4K is to be compared with theoretical limits, maybe i play with L2 frame size later etc.
fio shows definetly more on nfs server.

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 07.25.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4649
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1035.12
kills: 26022
deaths: 19948
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 810.059
kills: 3930
deaths: 3611
ratio: 1.08
setting MTU to 9000:
read bandwith iussue disappeared, for the price of a bit higher latency, resulting in ca. 10% IOPS loss:

Attachment:
Screenshot from 2021-07-25 20-20-51.png
Screenshot from 2021-07-25 20-20-51.png [ 25.77 KiB | Viewed 9462 times ]

Attachment:
Screenshot from 2021-07-25 20-21-05.png
Screenshot from 2021-07-25 20-21-05.png [ 42.51 KiB | Viewed 7744 times ]

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 08.09.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4649
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1035.12
kills: 26022
deaths: 19948
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 810.059
kills: 3930
deaths: 3611
ratio: 1.08
oki heres next try with nfs server...

Attachment:
Screenshot from 2021-08-09 19-41-26.png
Screenshot from 2021-08-09 19-41-26.png [ 889.19 KiB | Viewed 9417 times ]



++check the professional cooling holes hihi

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 08.11.21 
Offline
Godlike
User avatar

Joined: 03.12.11
Posts: 524
-----tdm:  
nick: [dswp]zietsh
skill: 1165.19
kills: 11410
deaths: 8841
ratio: 1.29
-----bomb:  
nick: [dswp]zietsh
skill: 1114.07
kills: 18153
deaths: 14802
ratio: 1.22
interesting, i feel like I need to update my vocabulary.

but i see lots of ssds, fast connections - i like it.



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 09.03.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4649
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1035.12
kills: 26022
deaths: 19948
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 810.059
kills: 3930
deaths: 3611
ratio: 1.08
UPDATE

...still nothing new from the disk.

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 10.05.21 
Offline
Godlike
User avatar

Joined: 08.31.08
Posts: 288
Location: 1408
-----tdm:  
nick: TitaniusAngleSmith
skill: 1355.62
kills: 10269
deaths: 7557
ratio: 1.35
-----bomb:  
nick: TitaniusAngleSmith
skill: 958.414
kills: 1010
deaths: 717
ratio: 1.40
wurst wrote:
UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 11.07.21 
Offline
Godlike

Joined: 03.29.10
Posts: 1450
-----tdm:  
nick: Pirat
skill: 465.668
kills: 402
deaths: 636
ratio: 0.63
bky wrote:
wurst wrote:
UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:


Deez nuts ... ;)


Pirat.



 


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 55 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Imprint

Powered by phpBB © 2000, 2002, 2005, 2007, 2008, 2009, 2010 phpBB Group

Skin by Lucas Kane
we use apache2 mod rewrite with phpBB SEO
map train_beta1-100 © 2009 by SteveMcQueen
Statistics Backend + Database by XLR Stats and B3 Bot
mapvote robot and gameserver monitor © 2009 by BlinKy
mumble viewer Copyright © 2008 Dominik Radner (aka Urmel)
mumble switcher and integration © 2008 by XTJ7, Unclefragger and Wursti
Localisation Plugin © 2009, Team Leads Plugin © 2009 and Knifer Plugin © 2009 by SvaRoX
for our stats we use Chart.js Copyright (c) 2013-2015 Nick Downie.
the stats also use some jQuery jQuery v2.1.4 | (c) 2005, 2015 jQuery Foundation, Inc.



voice server
bomb » dswp.de:22223
status: online
players: 0/24
map: ut4_mandolin
 
jump » dswp.de:22224
status: online
players: 0/20
map: q_derpjumps_final
 
freeze » dswp.de:22225
status: online
players: 0/32
map: ut4_casa
 
tdm » dswp.de:22226
status: online
players: 0/32
map: ut4_casa
 
tdm » dswp.de:22227
status: online
players: 0/32
map: ut4_turnpike
 
btw test the new ajax serwer monitor here!
top 20 players


nameskillkills
-Dws.BLINGBLING*-*1071.72418631
NormaSnockers1155.39400492
Zottel1090.4276378
make.them.suffer1143.27269872
>8v=1130.37230156
moon1100.78195615
sjas1048.1192315
peace1163.51190660
Wagner_Moura967.849188001
Goomba1151.68182677
z0rn996.032181016
Mad1117.01179124
[dswp]PLZ1144.31178516
Graf_ZahlIII1136.8167407
Zohan997.679159737
ubercunt1012.45159240
Yarrr!1187.33156233
I_am_nOOb1182.57151268
Pandageddon1171.49148319
Pirat1030.51145798