Re: apologies
We deployed the AO3 this weekend.
Since then![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.
Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.
But if you're nice, you apologize when people complain.
Well, I do not apologize for this deploy.
Who am I?
I'm the co-chair of OTW's System committee.
I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).
In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.
I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.
My actual quote before the deploy was:
If anyone is responsible for this code making it onto production, it is me.
And I do not apologize for this deploy.
This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.
There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.
That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.
But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.
None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.
So.
If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.
Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.
Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.
But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.
And we have nothing at all to apologize for.
Since then
lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.
But if you're nice, you apologize when people complain.
Well, I do not apologize for this deploy.
Who am I?
I'm the co-chair of OTW's System committee.
I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).
In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.
I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.
My actual quote before the deploy was:
i'm shaking in my boots, because last time was such a disaster, but i guess we're as ready as we ever will be
If anyone is responsible for this code making it onto production, it is me.
And I do not apologize for this deploy.
This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.
There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.
That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.
But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.
None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.
So.
If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.
Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.
Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.
But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.
And we have nothing at all to apologize for.
no subject
1. it is the cumulative effect. This isn't the first majorly buggy release but I have heard grumblings before about buggy releases on journals and such. People will get louder over time if they don't think they are heard.
2. people were already yelling. Everything is up to questioning now. So buggy releases got added into "we don't have to bite our lip and shut up about this anymore" pile. Depending on if you like the conversation happening or not, it is a good or bad thing.
3. AO3 is not an alpha or a closed beta release. It is an open beta, a very popular one. the change of volume from last time to this is in part a positive indication: people rely on the site rather than treat it like a weird experiment. If anything you guys should be proud, and accept the responsibility that comes with it rather than getting upset at the outrage. Also note that unlike packaged software, there is no way for users to use the old version of the code if for any reason the new version is buggy.
Also... I don't think individual coders need to apologize, or the Board. well okay Board apologizing is a nice touch. Point is, they shouldn't be apologizing because people are yelling. They should be apologizing due to (valid) concerns raised and mistakes WERE made. (in this case it is management issues). Most importantly when silence returns it shouldn't be back to good old ways, or to we'll fix this after the next deploy.
Just my 2 cents.