Re: apologies

[sidra]

We deployed the AO3 this weekend.

Since then lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.

Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.

But if you're nice, you apologize when people complain.

Well, I do not apologize for this deploy.

Who am I?

I'm the co-chair of OTW's System committee.

I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).

In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.

I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.

My actual quote before the deploy was:

i'm shaking in my boots, because last time was such a disaster, but i guess we're as ready as we ever will be

If anyone is responsible for this code making it onto production, it is me.

And I do not apologize for this deploy.

This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.

There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.

That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.

But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.

None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.

So.

If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.

Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.

Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.

But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.

And we have nothing at all to apologize for.

Comments

[mollyamory]

You really, really don't have anything to apologize for. *HUG* You're amazing, and the work you have undertaken for the org for SO LONG is amazing. I'm so glad you don't feel like you've done anything wrong - you haven't, and Lim didn't, and the people who think otherwise have absolutely no idea what goes into every release.

[sidra]

Thanks :)

[impertinence]

I'm not sure I've seen anyone asking for apologies, but I have seen a lot of requests for increased clarity and comprehensiveness. Is the AO3 code documented anywhere?

[sidra]

I'm not sure I've seen anyone asking for apologies

I haven't seen anyone asking for an apology either. This post was in response to the people who did apologize, not in response to anyone who might have asked for one.

I have seen a lot of requests for increased clarity and comprehensiveness

Skud's post is a wealth of useful constructive criticism and we'll get around to implementing as much of it as we can as soon as we can within the limitations and constraints that we work under. We set up the in-house volunteer system to help train people who don't know how to code. People who already know what they're doing can fork the code and submit pull requests. Which skud has already done. It's one of the main reasons we moved to github.

Is the AO3 code documented anywhere?

The AO3 code is documented in the code. It's not documented very well. If you don't want to wait for us to fix it and would like to fork it and improve its documentation yourself, please, feel free.

[impertinence]

I'm sorry, I was referring to independent documentation, similar to (though obviously not as complex as) Java's documentation. Another option is to have both official FAQs/documentation and community-driven documentation, the way Ubuntu does.

I'm not currently capable of writing such documentation due to time constraints, but I do think, as someone who's interested in doing driveby coding, that the more you guys attempt to recruit casual, experienced coders, the better off you'll be. I'd be happy to talk with someone about working on documentation in the future, though, if I knew there was a place to put it, and what the plans for community documentation were/are. :)

(sorry about the borked reply - that's what I get for making comments during finals week)

[sidra]

Unfortunately, you're not alone in not having the time at the moment. I didn't add Technical Writers to the people who aren't crawling out of the woodwork offering to help, but I could have. I know we've had a number of attempts at it in the past, but sadly they all fell by the wayside because of lack of time.

[impertinence]

Well, like I said, I'd be happy to make the time! But first I'd need to talk with people who'd tell me how/when my documentation would be published and utilized.

[elynross]

You, and all the OTW coders and support and what have you, are amazing. I think you should all be incredibly proud of what you've managed in a stunningly short period of time. And I think what you have to say here is AWESOME.

[sidra]

thanks :)

[onceamy]

This is not the first time (!) she has over-ruled other people's contributions, and hindered development because her pet project (Yuletide) must consume all the energies of AD&T.

Add Systems to that list, then. It was clear that you Systems people value the AO3 over other projects (notably Fanlore), but I did try to give you the benefit of the doubt.

Oh, but us meanies demand personal apologies, signed, notarized, and sent via hard copy! All I have ever asked for in this election cycle is respect for the work non-AO3 staff and volunteers are doing. I also asked that responsibility be taken by AD&T on rushing a release for Yuletide, and ignoring bug reports that were made by testers.

[sidra]

I have absolutely no idea what you're talking about.

This is not the first time (!) she has over-ruled other people's contributions, and hindered development because her pet project (Yuletide) must consume all the energies of AD&T.

I don't know what this quote means, or where it comes from. Or what it has to do with anything I said in my post.

Add Systems to that list, then. It was clear that you Systems people value the AO3 over other projects (notably Fanlore), but I did try to give you the benefit of the doubt.

I have no idea why you think systems values ao3 over fanlore, considering the huge amount of work we've put in lately on fanlore, and the emergency fixes we've put in just this week.

Oh, but us meanies demand personal apologies, signed, notarized, and sent via hard copy! All I have ever asked for in this election cycle is respect for the work non-AO3 staff and volunteers are doing.

This post had nothing at all to do with elections. It had to do with the last deploy and the fallout from it, specifically lim resigning and hele's apology from the board about the quality of the last deploy.

And as I said to impertinence, this post was not in response to people asking for apologies but in response to two specific apologies that were already made. I didn't know you *had* demanded an apology for the last deploy.

I also asked that responsibility be taken by AD&T on rushing a release for Yuletide, and ignoring bug reports that were made by testers.

I'm sorry you're unhappy about the deploy. I'm sorry you think we rushed a release. I'm sorry you think we're ignoring bug reports.

I stand by my statement. I don't agree that we rushed this release. If anything it was delayed. I don't agree that we ignored bug reports made by testers. You cannot delay releases indefinitely until you fix every bug.

[onceamy]

That quote is my own, taken from my latest post on the whole situation. I should have linked to it.

I will send you a private email regarding Fanlore, as I do not want to air out dirty laundry.

Also in my post is a personal recount of testing and finding the mobile skins bug in September, which I reported to a proxy. It was not fixed.

vague insinuatons are vague

[esteefee]

>I will send you a private email regarding Fanlore, as I do not want to air out dirty laundry.<

I'm sorry, but vague, unsupported insinuations don't stand. We don't have to be specific to be honest.

There are only two open Fanlore tickets. One of which Systems has been working on for days now fixing the various parts of, and the other of which was opened just the other day and only because I looped back on a dropped thread and urged you guys to open it, and is waiting for our Weds work meeting. (It's an install that might take some time.)

We respond to Fanlore tickets just as quickly as we do to any other tickets. I can honestly state without a twinge of discomfort that I treat all tickets the same. I am ticket agnostic; I really could give a damn who they are for. I'm more concerned with how critical they are for the continuing function of whatever system they impact. That's *all* I care about upon first glance. Is a system down? That ticket gets top priority. Is a system bogging down into near decrepitude? Definitely higher priority than an OJS upgrade. That's how a sysadmin thinks.

After that: easy tickets that take two minutes then get done fast to get them out of the way (we're talking volunteer intake tickets and the like).

Then we take on the others, the updates and the upgrades and the backups and the improvements and the maintenance and infrastructure discussions, time depending.

But you know that, Amy. Please remember there's a reason Systems uses tickets. It's a record of everything people have asked us to do, and it's a record of everything you have said to us, and that we have said in return. We have it all down in black and white.

Re: vague insinuatons are vague

[onceamy]

We don't have to be specific to be honest.

Actually, I do, because there have been situations where I have noticed a definite lack of regard for Fanlore. I cannot comment on Journal's systems, or anything running internally, as I do not use it enough, but I can comment specifically on Fanlore.

[willidan]

You're right. None of you have anything to apologize for. Personally, I think it looks awesome. I've been using it a handful of months now (got an invite this summer, I believe) and before that I read a handful of stories on there. But now I'm there nearly every day. I really appreciate all of your hard work. I have no idea what you do, but I know that it takes a lot of work and skill and you get nothing from it except pride in your accomplishments. You have every reason to be proud. And thank you again for all of your hard work.

[sidra]

thanks! Glad you like it :) It really cheered me up when i saw your post the other day.

[franzeska]

\o/

Coders should never have to apologize for there being some bugs; there are always some bugs. :)

I do think it's appropriate for the board to at least comment when enough people are angry because that's the sort of thing boards are there for, and it may be that there are timing and planning issues to address, but not working hard enough? Not spending tons of time? Not trying? No way. That's never been a problem.

[sidra]

Thanks :)

[coffeeandink]

Your protectiveness of your team is laudable. Your contempt for your user base is not.

Thanks for the transparency and openness, I guess.

[sidra]

I don't have contempt for the AO3 user base. I don't even know the vast majority of them.

I do think that people who use personal attacks and foul language to find fault, instead of calmly pointing out what they would like to see improved are mean. But I wouldn't say I have contempt for them.

[maraceles]

As part of that anonymous user base, I didn't think you showed contempt for us, and I'd just like to thank you for all your hard work.

[sheafrotherdon]

Thank you for your hard, hard work!

[sidra]

Thank you for saying so! :)

[autumnus]

there are 3 main reasons why people complained harder this time around.

1. it is the cumulative effect. This isn't the first majorly buggy release but I have heard grumblings before about buggy releases on journals and such. People will get louder over time if they don't think they are heard.

2. people were already yelling. Everything is up to questioning now. So buggy releases got added into "we don't have to bite our lip and shut up about this anymore" pile. Depending on if you like the conversation happening or not, it is a good or bad thing.

3. AO3 is not an alpha or a closed beta release. It is an open beta, a very popular one. the change of volume from last time to this is in part a positive indication: people rely on the site rather than treat it like a weird experiment. If anything you guys should be proud, and accept the responsibility that comes with it rather than getting upset at the outrage. Also note that unlike packaged software, there is no way for users to use the old version of the code if for any reason the new version is buggy.

Also... I don't think individual coders need to apologize, or the Board. well okay Board apologizing is a nice touch. Point is, they shouldn't be apologizing because people are yelling. They should be apologizing due to (valid) concerns raised and mistakes WERE made. (in this case it is management issues). Most importantly when silence returns it shouldn't be back to good old ways, or to we'll fix this after the next deploy.

Just my 2 cents.

[rivkat]

I just wanted to say thanks. I am often siloed in my specific area, and I know I send you guys silly tickets, and I don't even notice because Systems works so hard and so well. My feelings: We're facing the curse of success, because we look big from the outside, but we certainly don't get stock options or time off from the day jobs many of us have! I hope in the coming year I can do my part to help us grow, but I know it's not as big as your part!