Re: apologies
We deployed the AO3 this weekend.
Since then![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.
Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.
But if you're nice, you apologize when people complain.
Well, I do not apologize for this deploy.
Who am I?
I'm the co-chair of OTW's System committee.
I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).
In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.
I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.
My actual quote before the deploy was:
If anyone is responsible for this code making it onto production, it is me.
And I do not apologize for this deploy.
This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.
There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.
That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.
But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.
None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.
So.
If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.
Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.
Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.
But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.
And we have nothing at all to apologize for.
Since then
lim has apologized and resigned. I don't believe she has anything to apologize for, but I don't blame her for resigning. It's demoralizing to see your hard work torn to shreds by people who have no idea of the constraints we work under. Not just the constraints of time and personnel, but the constraints of more than fifty different parsers and many complex accessibility requirements. And nobody wants to work somewhere where their work will be hated so much that people will rudely and publicly vilify it.Hele has apologized on behalf of the board and promised to make sure it never happens again. I don't think the board has anything to apologize for either, unless they were the cause of the vitriol, which as far as I know is not the case. But I do think their promise is unrealistic. No matter what they do, they'll never be able to make sure it never happens again. In twenty-five years I have not once seen a deploy in RL which was bug free and didn't cause a single complaint from the users. And that's when there are full-time highly paid developers, highly paid release engineers, and very highly paid project managers. All of whom have stock options which will be worth nothing unless they make a success of the company and so are extremely vested in the outcome. No matter how much we improve the process or the code, we will never be able to produce a 100% bug-free deploy that everyone loves. It's not possible. Honest, it's really just not possible.
But if you're nice, you apologize when people complain.
Well, I do not apologize for this deploy.
Who am I?
I'm the co-chair of OTW's System committee.
I'm a senior coder for the archive and one of the owner's of the archive code on github. (ambtus).
In RL I'm a senior unix system administrator. If you're interested in my professional qualifications, check out my public Linkedin profile. I'm not a professional developer, but I have twenty-five years experience working with or in development departments. First in QA, and then as a sysadmin where I work closely with development and release engineering.
I wrote the code that pushes the code to production. I made the changes on the server which were necessary for the new code to be deployed. I control access to the servers for the people who push the code. I trained and encouraged and enabled the person who did the push this time.
My actual quote before the deploy was:
i'm shaking in my boots, because last time was such a disaster, but i guess we're as ready as we ever will be
If anyone is responsible for this code making it onto production, it is me.
And I do not apologize for this deploy.
This deploy, despite receiving an extraordinary amount of negative feedback (and an unusually large amount of cruel and hurtful feedback in the form of personal attacks against the developers), was not more buggy than usual, or worse than usual, or more than typically bad.
There have been other deploys that melted the servers. Deploys that had huge security bugs. Deploys which broke major functionality. Deploys which were a great deal worse than this one. I worked for twenty-four hours straight last time to recover from the last deploy because we did something to the code which melted the database server.
That's not to say the current code should be lauded just because it's better than what we had before. Not at all. I know more than anyone just how much work we still have to do. Believe me, we know what the problems are. We're not ignoring the support requests or the tweets or the LJ posts. We don't think our process is perfect. For that matter, I don't even think our process is very good but we're continually working to improve it.
But we don't have anything to apologize for. We're doing our best under very difficult circumstances. Developers are not crawling out of the woodwork begging to be allowed to work on this project. We don't have release engineers and project managers waiting in the wings to help us. But the users are lining up in droves to join and usage just continues to increase by leaps and bounds.
None of the people who worked on this release have anything to apologize for. The board does not need to apologize for not having stepped in to prevent it from happening or promise it will never happen again.
So.
If you don't like something we do, please open a support ticket. If you're nice about it we'll do our best to fix it as soon as we can.
Actually, we'll do our best to fix it even if you're rude and hateful, because our support volunteers are a lot nicer than I am and will pass on your issues (although not your vitriol) to the developers no matter what.
Or fix it yourself. We put the code on github so that anyone can fork it and send us a pull request. We'll be more than grateful for anything we can use.
But don't ask me to apologize for what we've produced so far. Because it's an amazing and complex piece of work, completely coded from scratch by amateurs and volunteers.
And we have nothing at all to apologize for.
no subject
I will send you a private email regarding Fanlore, as I do not want to air out dirty laundry.
Also in my post is a personal recount of testing and finding the mobile skins bug in September, which I reported to a proxy. It was not fixed.
vague insinuatons are vague
I'm sorry, but vague, unsupported insinuations don't stand. We don't have to be specific to be honest.
There are only two open Fanlore tickets. One of which Systems has been working on for days now fixing the various parts of, and the other of which was opened just the other day and only because I looped back on a dropped thread and urged you guys to open it, and is waiting for our Weds work meeting. (It's an install that might take some time.)
We respond to Fanlore tickets just as quickly as we do to any other tickets. I can honestly state without a twinge of discomfort that I treat all tickets the same. I am ticket agnostic; I really could give a damn who they are for. I'm more concerned with how critical they are for the continuing function of whatever system they impact. That's *all* I care about upon first glance. Is a system down? That ticket gets top priority. Is a system bogging down into near decrepitude? Definitely higher priority than an OJS upgrade. That's how a sysadmin thinks.
After that: easy tickets that take two minutes then get done fast to get them out of the way (we're talking volunteer intake tickets and the like).
Then we take on the others, the updates and the upgrades and the backups and the improvements and the maintenance and infrastructure discussions, time depending.
But you know that, Amy. Please remember there's a reason Systems uses tickets. It's a record of everything people have asked us to do, and it's a record of everything you have said to us, and that we have said in return. We have it all down in black and white.
Re: vague insinuatons are vague
Actually, I do, because there have been situations where I have noticed a definite lack of regard for Fanlore. I cannot comment on Journal's systems, or anything running internally, as I do not use it enough, but I can comment specifically on Fanlore.