Gameservers Down

Description lol

Gameservers Down

Postby wurst » 07.24.21

hi everyone...
last days the game servers were down, its cause a disk died and there was no backup.
actually it was a mobile disk, usb, originally "temporarily" connected (and forgotten) over the last 10 years.
vmware doesnt even support usb drives natively, this setup was in fact left over from a migration.
hm.

the drive:
Screenshot from 2021-07-24 07-34-11.png
(7.48 KiB) Downloaded 4600 times


so i thought about 3 things.
- recovering the data from the disk and, ofc,
- getting a proper store and
- backup.

##### PART 1 RECOVERY #####

i dont have a car at the moment, the datacenter is some hours by train away.
blinky lives close there, so he went to datacenter to get that disk.
i myself bought some hardware and sent it to blinky before, so he could pimp the storage hw a bit.

result of blinkys inspection:
the disk comes up when connected.
gparted shows an unknown partition, but its vmware vmfs, thats normal:
Screenshot from 2021-07-24 07-43-46.png


but it fails then after some seconds. The warmer it gets, the faster it disconnects.
It disconnects usb-device-wise, so at least the drive seems ok.

now blinky had an idea.
this is the drive:
Screenshot from 2021-07-24 07-46-24.png
Screenshot from 2021-07-24 07-46-34.png


the board is a 2060-771801-002 REV A
it looks like swapping a board from an identical disk could bring the disk back.
but not the data. partitioning informations are stuck in the bios of the old hard disk.
fortunately, blinky got the soldering master degree, so we ordered a board...

this is todays state of the project, blinky swapped already the bios chips.
Screenshot from 2021-07-24 07-51-51.png


##### PART 2 NEW DISK #####

1.) Hardware...
a year ago, i started to build some storage.
basically, it was cause there was some time and money.
so i bought a dell 730xd as a first storage box...
Screenshot from 2021-07-24 08-11-38.png


and i got a pair of juniper ex4200 switches, so i could run ha storage and ip networks.
Screenshot from 2021-07-24 08-24-21.png


the 730xd is ideal imho, it has 25 sas slots and alot pcie slots/lanes for nvme.
the ex4200 have each 2x 10g uplinks, the dell server has 2x 10g.
so the switches have space for 2x storage with 2x 10g each.

anyhow, the hypervisors must be connected, too and theres jist 1g links left.
the ex4200 can aggregate 8 physical links, so thats what i planned:
every hv gets 8 links, there will be a dswitch with lacp in vmware.
additionally, the 730xd needs some cheap test disks...
here we are. intenso 240gb.
normally a bad choice but bigul checked them ok.
Screenshot from 2021-07-24 08-27-36.png

testing the packaging hardware for transport!
Screenshot from 2021-07-24 08-27-55.png
(378.74 KiB) Downloaded 4600 times


when blinky was in datacenter, he could mount nics and ssds.
so it looks like that now:
Screenshot from 2021-07-24 08-34-53.png
(91.39 KiB) Downloaded 4599 times

Screenshot from 2021-07-24 08-36-09.png


dswitch and lacp are on, next step will be playing with nfs4, vmware and multipathing...
Screenshot from 2021-07-24 08-38-50.png


expect an update sooner or later. (:
anyone interested in storage and ethernet?
Image
User avatar
wurst
Godlike
 
Posts: 4648
Joined: 07.15.08
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09

Re: Gameservers Down

Postby wurst » 07.24.21

in vmware, theres several ways to connect storage.
for redundant connections, fibre channel ist widely known.
smaller installs may use iscsi or even old nfs.

now comes my ideas...
fibrechannel is no option, plus i think that block based storage belongs to local chassis.
vmware dont need block devices really.
the setup should combine storage and ip networking, saves alot space and power.

iscsi is the more modern way to go ip based.
but its block based, i wanted to get rid of this.

look how old nfs does true active/active multipathing:
Screenshot from 2021-07-24 11-29-25.png


the result is an extra 0,2 ms access time to the intenso lol ssd, as seen from a windows vm:
Screenshot from 2021-07-24 11-19-47.png

Screenshot from 2021-07-24 11-27-01.png
(20.45 KiB) Downloaded 4591 times


multipathing works through IP-hash-based link aggregation and nfs 4 does IP- based multipathing.
if you dont have a switch that can do link aggregation:
take 10 miniswitches, 10 network cards in each server, 10 subnets and you go.
you can even combine fast and slow links, which is one of many advantages of IP.

the other advantage is the file system.
in vmware, you cant choose a filesystem, theres only "vmfs"
recovering this failed disk means mounting it in vmware.
now i can have ext, xfs, i have even ram for zfs in the 730xd...

for the first i will track that read bandwith thingy, looks a bit mad with 200MB/s.
30K IOPS @ 4K is to be compared with theoretical limits, maybe i play with L2 frame size later etc.
fio shows definetly more on nfs server.
Image
User avatar
wurst
Godlike
 
Posts: 4648
Joined: 07.15.08
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09

Re: Gameservers Down

Postby wurst » 07.25.21

setting MTU to 9000:
read bandwith iussue disappeared, for the price of a bit higher latency, resulting in ca. 10% IOPS loss:

Screenshot from 2021-07-25 20-20-51.png

Screenshot from 2021-07-25 20-21-05.png
(42.51 KiB) Downloaded 4582 times
Image
User avatar
wurst
Godlike
 
Posts: 4648
Joined: 07.15.08
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09

Re: Gameservers Down

Postby wurst » 08.09.21

oki heres next try with nfs server...

Screenshot from 2021-08-09 19-41-26.png



++check the professional cooling holes hihi
Image
User avatar
wurst
Godlike
 
Posts: 4648
Joined: 07.15.08
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09

Re: Gameservers Down

Postby zietsh » 08.11.21

interesting, i feel like I need to update my vocabulary.

but i see lots of ssds, fast connections - i like it.
User avatar
zietsh
Godlike
 
Posts: 524
Joined: 03.12.11
-----tdm:  
nick: [dswp]zietsh
skill: 991.948
kills: 11203
deaths: 8618
ratio: 1.29
-----bomb:  
nick: [dswp]zietsh
skill: 1056.62
kills: 18077
deaths: 14755
ratio: 1.22

Re: Gameservers Down

Postby wurst » 09.03.21

UPDATE

...still nothing new from the disk.
Image
User avatar
wurst
Godlike
 
Posts: 4648
Joined: 07.15.08
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09

Re: Gameservers Down

Postby bky » 10.05.21

wurst wrote:UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:
Image
User avatar
bky
Godlike
 
Posts: 285
Joined: 08.31.08
Location: 1408
-----tdm:  
nick: TitaniusAngleSmith
skill: 1164.22
kills: 9429
deaths: 6605
ratio: 1.42
-----bomb:  
nick: TitaniusAngleSmith
skill: 886.644
kills: 963
deaths: 678
ratio: 1.42

Re: Gameservers Down

Postby Pirat » 11.07.21

bky wrote:
wurst wrote:UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:


Deez nuts ... ;)


Pirat.
Pirat
Godlike
 
Posts: 1447
Joined: 03.29.10
-----tdm:  
nick: Pirat
skill: 507.365
kills: 402
deaths: 636
ratio: 0.63


Who is online

Users browsing this forum: Google [Bot] and 22 guests

Misc