\
  The most prestigious law school admissions discussion board in the world.
BackRefresh Options Favorite

Amazon outage was due to an engineer entering a typo on the command line

At 9:37AM PST, an authorized S3 team member using an establi...
angry rough-skinned orchestra pit
  03/03/17
automation can't come soon enough
At-the-ready Toaster
  03/03/17
Somewhere down the line there still has to be a dude running...
angry rough-skinned orchestra pit
  03/03/17
Ah crapnuts, I just activated the robots in "experiment...
hideous zombie-like double fault
  03/03/17
there are automation systems like Chef, Puppet, Ansible that...
Transparent Appetizing Shrine Community Account
  03/03/17
It just means you can use one sysadmin for huge clusters ins...
hideous zombie-like double fault
  03/03/17
at least in primary markets, I don't think sysadmins really ...
Transparent Appetizing Shrine Community Account
  03/03/17
There are definitely still sysadmins but they need to know h...
hideous zombie-like double fault
  03/03/17
right, the DevOps thing is what I meant, or "infrastruc...
Transparent Appetizing Shrine Community Account
  03/03/17
(Not Skynet)
Glittery Jade Private Investor
  03/03/17
rm -rf /
obsidian infuriating juggernaut
  03/03/17
sounds like the plot of a Futurama episode
hideous zombie-like double fault
  03/03/17
*urkel voice* Did I do thaaaat?
dashing fat ankles stock car
  03/03/17
COMMA CHECKER JD's w biglaw exp will always be hot commoditi...
gold galvanic azn abode
  03/03/17
Compliance Counsel of Typography
Learning Disabled Abusive Striped Hyena
  03/03/17
...
angry rough-skinned orchestra pit
  03/03/17
this sounds like a cover-story for getting hacked. it's lik...
green thirsty psychic
  03/03/17
yup
Transparent Appetizing Shrine Community Account
  03/03/17
likely the russians.
pearl gaping parlor
  03/03/17
Yeah, probably payback for WaPo trashing their boy.
blue sexy heaven ladyboy
  03/03/17
Command line is inefficient and terrible. Why don't they jus...
Slate affirmative action philosopher-king
  03/03/17
command line is far more efficient than a GUI
Transparent Appetizing Shrine Community Account
  03/03/17
GUI usually means you need mouse support/drivers, and that s...
green thirsty psychic
  03/03/17
I think the fact that some devops dude took down the interne...
angry rough-skinned orchestra pit
  03/03/17
(knows shit about programming)
pearl gaping parlor
  03/03/17
(not a command line MFE)
Thriller Primrose Regret Main People
  03/03/17
I guess the follow-up here is whether this engineer still ha...
angry rough-skinned orchestra pit
  03/03/17


Poast new message in this thread



Reply Favorite

Date: March 3rd, 2017 6:05 PM
Author: angry rough-skinned orchestra pit

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems.

https://aws.amazon.com/message/41926/

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746468)



Reply Favorite

Date: March 3rd, 2017 6:06 PM
Author: At-the-ready Toaster

automation can't come soon enough

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746474)



Reply Favorite

Date: March 3rd, 2017 6:10 PM
Author: angry rough-skinned orchestra pit

Somewhere down the line there still has to be a dude running commands like that though.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746487)



Reply Favorite

Date: March 3rd, 2017 6:11 PM
Author: hideous zombie-like double fault

Ah crapnuts, I just activated the robots in "experimental sentience" mode. *cracks open beer and rolls back over in bed*

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746493)



Reply Favorite

Date: March 3rd, 2017 6:13 PM
Author: Transparent Appetizing Shrine Community Account

there are automation systems like Chef, Puppet, Ansible that can be programmed to perform corrective actions on infrastructure based on certain triggers and conditions. though ultimately there needs to be some human element at the end of the line.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746500)



Reply Favorite

Date: March 3rd, 2017 6:39 PM
Author: hideous zombie-like double fault

It just means you can use one sysadmin for huge clusters instead of small ones ultimately. Given the cloudsplosion it's basically a net neutral in how many of these dudes are necessary.

They do need increasingly specialized skillets though. 15 years ago a doofus who knew how to load a Linux CD could get a 50k sysadmin job in low col area.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746645)



Reply Favorite

Date: March 3rd, 2017 6:49 PM
Author: Transparent Appetizing Shrine Community Account

at least in primary markets, I don't think sysadmins really exist anymore. there are engineers who code applications and there are engineers who code infrastructure, and the latter will be able to (and be expected to) do anything a sysadmin could. either that or a highly specialized role that requires deep understanding of Linux internals, C, strace, etc.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746704)



Reply Favorite

Date: March 3rd, 2017 6:55 PM
Author: hideous zombie-like double fault

There are definitely still sysadmins but they need to know how to do a lot more stuff and have generally been rebadged as "DevOps/Systems Engineers/etc.". They need to know how to code and identify bugs but don't usually work on infrastructure products directly, and they have generally branched off into more specialized roles at bigcorp (e.g. guys who do only networking, etc). There are still generalists left at smaller places like academic or in-house computing clusters as well though

It's not like you have guys who develop AWS services also doing the work to install and keep everything running. That would be pretty inefficient and chaotic

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746729)



Reply Favorite

Date: March 3rd, 2017 7:00 PM
Author: Transparent Appetizing Shrine Community Account

right, the DevOps thing is what I meant, or "infrastructure as code". that usually means high-level languages scripting the deployment of cloud formations, with some kind of CMS monitoring the config states and automating changes. every now and then you may need to fire off a manual command, which is what the traditional sysadmin used to do.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746758)



Reply Favorite

Date: March 3rd, 2017 7:08 PM
Author: Glittery Jade Private Investor

(Not Skynet)

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746796)



Reply Favorite

Date: March 3rd, 2017 6:07 PM
Author: obsidian infuriating juggernaut

rm -rf /

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746475)



Reply Favorite

Date: March 3rd, 2017 6:10 PM
Author: hideous zombie-like double fault

sounds like the plot of a Futurama episode

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746489)



Reply Favorite

Date: March 3rd, 2017 6:24 PM
Author: dashing fat ankles stock car

*urkel voice*

Did I do thaaaat?

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746549)



Reply Favorite

Date: March 3rd, 2017 6:26 PM
Author: gold galvanic azn abode

COMMA CHECKER JD's w biglaw exp will always be hot commodities.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746565)



Reply Favorite

Date: March 3rd, 2017 6:47 PM
Author: Learning Disabled Abusive Striped Hyena

Compliance Counsel of Typography

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746687)



Reply Favorite

Date: March 3rd, 2017 11:41 PM
Author: angry rough-skinned orchestra pit



(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32748578)



Reply Favorite

Date: March 3rd, 2017 6:49 PM
Author: green thirsty psychic

this sounds like a cover-story for getting hacked. it's like back in the 80's when poorly-placed unix commands would be blamed for causing shit that was actually being done via network infiltration.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746702)



Reply Favorite

Date: March 3rd, 2017 6:50 PM
Author: Transparent Appetizing Shrine Community Account

yup

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746707)



Reply Favorite

Date: March 3rd, 2017 7:10 PM
Author: pearl gaping parlor

likely the russians.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746805)



Reply Favorite

Date: March 3rd, 2017 11:50 PM
Author: blue sexy heaven ladyboy

Yeah, probably payback for WaPo trashing their boy.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32748604)



Reply Favorite

Date: March 3rd, 2017 7:09 PM
Author: Slate affirmative action philosopher-king

Command line is inefficient and terrible. Why don't they just do GUI bro?

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746801)



Reply Favorite

Date: March 3rd, 2017 7:16 PM
Author: Transparent Appetizing Shrine Community Account

command line is far more efficient than a GUI

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746856)



Reply Favorite

Date: March 3rd, 2017 7:16 PM
Author: green thirsty psychic

GUI usually means you need mouse support/drivers, and that starts getting into more than just a bare-bones interface and into operating systems. command line can run off of a bare motherboard with no complex OS.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746860)



Reply Favorite

Date: March 3rd, 2017 7:17 PM
Author: angry rough-skinned orchestra pit

I think the fact that some devops dude took down the internet on one command shows that it actually is pretty efficient.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746870)



Reply Favorite

Date: March 3rd, 2017 7:17 PM
Author: pearl gaping parlor

(knows shit about programming)

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746873)



Reply Favorite

Date: March 3rd, 2017 7:19 PM
Author: Thriller Primrose Regret Main People

(not a command line MFE)

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746888)



Reply Favorite

Date: March 3rd, 2017 7:19 PM
Author: angry rough-skinned orchestra pit

I guess the follow-up here is whether this engineer still has a job.

(http://www.autoadmit.com/thread.php?thread_id=3542528&forum_id=2#32746882)