sidra: mandala (Default)
Sidra ([personal profile] sidra) wrote2011-11-14 08:47 pm

Re: apologies

We deployed the AO3 this weekend.

Since then [personal profile] lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.

Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.

But if you're nice, you apologize when people complain.

Well, I do not apologize for this deploy.

Who am I?

I'm the co-chair of OTW's System committee.

I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).

In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.

I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.

My actual quote before the deploy was:

i'm shaking in my boots, because last time was such a disaster, but i guess we're as ready as we ever will be

If anyone is responsible for this code making it onto production, it is me.

And I do not apologize for this deploy.

This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.

There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.

That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.

But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.

None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.


If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.

Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.

Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.

But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.

And we have nothing at all to apologize for.
mollyamory: SPN - Impala going green (Default)

[personal profile] mollyamory 2011-11-15 02:25 am (UTC)(link)
You really, really don't have anything to apologize for. *HUG* You're amazing, and the work you have undertaken for the org for SO LONG is amazing. I'm so glad you don't feel like you've done anything wrong - you haven't, and Lim didn't, and the people who think otherwise have absolutely no idea what goes into every release.
impertinence: (Default)

[personal profile] impertinence 2011-11-15 04:41 am (UTC)(link)
I'm not sure I've seen anyone asking for apologies, but I have seen a lot of requests for increased clarity and comprehensiveness. Is the AO3 code documented anywhere?
impertinence: (Default)

[personal profile] impertinence 2011-11-15 07:51 pm (UTC)(link)
I'm sorry, I was referring to independent documentation, similar to (though obviously not as complex as) Java's documentation. Another option is to have both official FAQs/documentation and community-driven documentation, the way Ubuntu does.

I'm not currently capable of writing such documentation due to time constraints, but I do think, as someone who's interested in doing driveby coding, that the more you guys attempt to recruit casual, experienced coders, the better off you'll be. I'd be happy to talk with someone about working on documentation in the future, though, if I knew there was a place to put it, and what the plans for community documentation were/are. :)

(sorry about the borked reply - that's what I get for making comments during finals week)
impertinence: (Default)

[personal profile] impertinence 2011-11-16 01:35 am (UTC)(link)
Well, like I said, I'd be happy to make the time! But first I'd need to talk with people who'd tell me how/when my documentation would be published and utilized.
elynross: (Default)

[personal profile] elynross 2011-11-15 06:44 am (UTC)(link)
You, and all the OTW coders and support and what have you, are amazing. I think you should all be incredibly proud of what you've managed in a stunningly short period of time. And I think what you have to say here is AWESOME.
onceamy: Nothing special; just a pixelated rainbow. (Default)

[personal profile] onceamy 2011-11-15 01:23 pm (UTC)(link)
This is not the first time (!) she has over-ruled other people's contributions, and hindered development because her pet project (Yuletide) must consume all the energies of AD&T.

Add Systems to that list, then. It was clear that you Systems people value the AO3 over other projects (notably Fanlore), but I did try to give you the benefit of the doubt.

Oh, but us meanies demand personal apologies, signed, notarized, and sent via hard copy! All I have ever asked for in this election cycle is respect for the work non-AO3 staff and volunteers are doing. I also asked that responsibility be taken by AD&T on rushing a release for Yuletide, and ignoring bug reports that were made by testers.
onceamy: Nothing special; just a pixelated rainbow. (Default)

[personal profile] onceamy 2011-11-15 02:43 pm (UTC)(link)
That quote is my own, taken from my latest post on the whole situation. I should have linked to it.

I will send you a private email regarding Fanlore, as I do not want to air out dirty laundry.

Also in my post is a personal recount of testing and finding the mobile skins bug in September, which I reported to a proxy. It was not fixed.
esteefee: (talky-hands)

vague insinuatons are vague

[personal profile] esteefee 2011-11-16 12:50 am (UTC)(link)
>I will send you a private email regarding Fanlore, as I do not want to air out dirty laundry.<

I'm sorry, but vague, unsupported insinuations don't stand. We don't have to be specific to be honest.

There are only two open Fanlore tickets. One of which Systems has been working on for days now fixing the various parts of, and the other of which was opened just the other day and only because I looped back on a dropped thread and urged you guys to open it, and is waiting for our Weds work meeting. (It's an install that might take some time.)

We respond to Fanlore tickets just as quickly as we do to any other tickets. I can honestly state without a twinge of discomfort that I treat all tickets the same. I am ticket agnostic; I really could give a damn who they are for. I'm more concerned with how critical they are for the continuing function of whatever system they impact. That's *all* I care about upon first glance. Is a system down? That ticket gets top priority. Is a system bogging down into near decrepitude? Definitely higher priority than an OJS upgrade. That's how a sysadmin thinks.

After that: easy tickets that take two minutes then get done fast to get them out of the way (we're talking volunteer intake tickets and the like).

Then we take on the others, the updates and the upgrades and the backups and the improvements and the maintenance and infrastructure discussions, time depending.

But you know that, Amy. Please remember there's a reason Systems uses tickets. It's a record of everything people have asked us to do, and it's a record of everything you have said to us, and that we have said in return. We have it all down in black and white.
onceamy: Nothing special; just a pixelated rainbow. (Default)

Re: vague insinuatons are vague

[personal profile] onceamy 2011-11-16 04:34 am (UTC)(link)
We don't have to be specific to be honest.

Actually, I do, because there have been situations where I have noticed a definite lack of regard for Fanlore. I cannot comment on Journal's systems, or anything running internally, as I do not use it enough, but I can comment specifically on Fanlore.
Edited 2011-11-16 04:35 (UTC)
willidan: (Default)

[personal profile] willidan 2011-11-15 03:06 pm (UTC)(link)
You're right. None of you have anything to apologize for. Personally, I think it looks awesome. I've been using it a handful of months now (got an invite this summer, I believe) and before that I read a handful of stories on there. But now I'm there nearly every day. I really appreciate all of your hard work. I have no idea what you do, but I know that it takes a lot of work and skill and you get nothing from it except pride in your accomplishments. You have every reason to be proud. And thank you again for all of your hard work.
franzeska: (Default)

[personal profile] franzeska 2011-11-15 06:07 pm (UTC)(link)

Coders should never have to apologize for there being some bugs; there are always some bugs. :)

I do think it's appropriate for the board to at least comment when enough people are angry because that's the sort of thing boards are there for, and it may be that there are timing and planning issues to address, but not working hard enough? Not spending tons of time? Not trying? No way. That's never been a problem.
coffeeandink: (Default)

[personal profile] coffeeandink 2011-11-15 11:59 pm (UTC)(link)
Your protectiveness of your team is laudable. Your contempt for your user base is not.

Thanks for the transparency and openness, I guess.

[personal profile] maraceles 2011-11-16 02:50 am (UTC)(link)
As part of that anonymous user base, I didn't think you showed contempt for us, and I'd just like to thank you for all your hard work.
sheafrotherdon: (Default)

[personal profile] sheafrotherdon 2011-11-16 02:16 am (UTC)(link)
Thank you for your hard, hard work!
autumnus: A purple monochrome portrait of Zoe from Dreamfall, with drawn stars in background and "the Dreamer" written on bottom. (Default)

[personal profile] autumnus 2011-11-16 08:18 pm (UTC)(link)
there are 3 main reasons why people complained harder this time around.

1. it is the cumulative effect. This isn't the first majorly buggy release but I have heard grumblings before about buggy releases on journals and such. People will get louder over time if they don't think they are heard.

2. people were already yelling. Everything is up to questioning now. So buggy releases got added into "we don't have to bite our lip and shut up about this anymore" pile. Depending on if you like the conversation happening or not, it is a good or bad thing.

3. AO3 is not an alpha or a closed beta release. It is an open beta, a very popular one. the change of volume from last time to this is in part a positive indication: people rely on the site rather than treat it like a weird experiment. If anything you guys should be proud, and accept the responsibility that comes with it rather than getting upset at the outrage. Also note that unlike packaged software, there is no way for users to use the old version of the code if for any reason the new version is buggy.

Also... I don't think individual coders need to apologize, or the Board. well okay Board apologizing is a nice touch. Point is, they shouldn't be apologizing because people are yelling. They should be apologizing due to (valid) concerns raised and mistakes WERE made. (in this case it is management issues). Most importantly when silence returns it shouldn't be back to good old ways, or to we'll fix this after the next deploy.

Just my 2 cents.
rivkat: otw searchlight in Batman signal form (otw searchlight)

[personal profile] rivkat 2011-11-19 03:01 am (UTC)(link)
I just wanted to say thanks. I am often siloed in my specific area, and I know I send you guys silly tickets, and I don't even notice because Systems works so hard and so well. My feelings: We're facing the curse of success, because we look big from the outside, but we certainly don't get stock options or time off from the day jobs many of us have! I hope in the coming year I can do my part to help us grow, but I know it's not as big as your part!