Karmic Cascade

To break up the usual code-filled technical blogs you see all day, we’ve got something a little different for you today. Our friend Remy Porter from The Daily WTF has written a guest post about the power of teamwork. You can read more of Remy’s work at The Daily WTF, and you can find him on Twitter too. Enjoy!

George didn’t exactly enjoy his job. He was a senior developer, doing software for a credit reporting and collections agency, which was neither “sexy” nor stimulating. He had some great co-workers, though, like his junior developer, Elaine, or the head of IT, Mr. Thomassulo.

Unfortunately, no number of great co-workers could make up for having to work with Lloyd. Lloyd was another senior developer, and Lloyd was a problem. Lloyd wasn’t just a senior developer, he had once been the IT director of a major hospital system. This was a fact he liked to repeat, ad nauseum: “I know what I’m doing,” he would say. “I used to run IT for six hospitals. I supported thousands of users, and it was life or death. I know what I’m doing.”

He knew what he was doing so well that he eschewed the organizational coding conventions. He knew what he was doing so well, that he would unilaterally make changes to applications, without discussing it with other developers, management, or the end users. When functionality inevitably broke as a result of his meddling, it was the extant code that was wrong, because “I know what I’m doing.” He never met a new library or API that he didn’t hate, because he already knew what he was doing, and any changes to his process were bad. At one point, he broke a turn-key, purchased application by cracking open the database and modifying the underlying schema, because “I know what I’m doing.”

The work was boring, but it was really Lloyd who put George over the edge. Constantly fighting with Lloyd, constantly cleaning up Lloyd’s messes, and constantly being undercut by Lloyd in front of management soured him. George started planning a change of job, but before he could leave, he was given One Last Project.

The project itself was the sort of thing that breeds quietly inside of “enterprise organizations”. Their customers- mostly property management and utility companies- needed to constantly exchange legal documents with George’s company. These documents had a processing workflow. There were plenty of applications they could buy which would solve this problem out of the box, but the combination of “not invented here”, “our needs are too specific for a generic product”, and “our capital budget is empty, but our development budget is full”, meant that they were building the application in-house.

George was the lead on the project, but George wasn’t giving it his full attention- he was trying to transition to a job that didn’t have a Lloyd. Of course, before he even saw a requirements document, the project was six months behind schedule. A bunch of middle managers had decided to hitch their wagons to this single horse, and that meant meddling. “Let’s set up a new CI server for just this project!” “Let’s switch to using Entity Framework, even though nobody at our organization has actually looked at the technology yet!” “We should do Ajax!” “The PMO has decided we will use a new methodology on this project, which we’re calling Scrum.” “I know our internal browser standard is IE8, but we’re going to do this in HTML5!”

The result was not George’s best work. The code could have been cleaner, the source control history could have been much cleaner, and he made a mistake. Entity Framework, when used in the “Code First” style, could generate your database schema from your object model. Something neither George nor Elaine discovered until the product was released, was that by default, it would enable cascading deletes on certain types of relationship. Specifically, there was a cascading relationship from entities in the StatusCode table to the documents which depended on those status codes.

George did discover his mistake, and so he did the best thing he could on his way out the door. He told Elaine, he documented the flaw, and he told Mr. Thomassulo, and gave him suggestions for fixing it. They could fix- or not fix- the flaw at their leisure. Since the application had no functionality for deleting status codes, Mr. Thomassulo made the call that this was a “don’t fix”.

George smiled, nodded, and moved on to his next job. Eight months went by, and George spent those months with people who weren’t Lloyd. He kept in touch with Elaine, exchanged the odd email with Mr. Thomassulo, but mostly thought that phase of his career was over.

Until, one afternoon, Elaine called his cellphone. “I am so happy, and so upset with you right now.”

Lloyd’s wanderings through the company’s codebase eventually brought him to the document management application. Lloyd saw that one of the status-codes was named, “Pre-Approval”, and since he ran IT for six hospitals, and knew what he was doing, he knew that the status code was wrong. He knew that it should be, “Waiting for approval…”. He didn’t need to check this over with the users, since he knew what he was doing. He didn’t need to discuss this with the other developers, since he knew what he was doing. He didn’t need to read the documentation, because he knew what he was doing. And he certainly didn’t need to pilot his changes in test, because he knew what he was doing.

If Lloyd had just done the simple thing: UPDATE StatusCodes SET Name='Waiting for Approval…' WHERE id=5, he would have had no problems. That’s what the documentation George had left behind recommended. But Lloyd knew what he was doing, and chose instead to DELETE the old status code, then INSERT a fresh one, then UPDATE the broken records. Of course, he didn’t get past the DELETE step before everything went terribly, terribly wrong. Half of the documents the users were actively working with vanished from the database.

“So,” Elaine explained, “I’m furious with you, because I got roped in to help Lloyd with fixing this. But I got smart- I offered to take on Lloyd’s other work, so that he could focus on this issue.”

Lloyd, of course, knew what he was doing, and thus didn’t need any help. Lloyd saw the problem of missing data, saw that the documents still were actually stored in a network share, saw that with some of the supporting meta-data tables, he could reconstruct the lost data using inference. So that’s what he did. Manually. After all, he knew what he was doing.

No one felt the need to stop him. Instead, Elaine and Mr. Thomassulo looked on, in awe, as Lloyd brute-forced his way to insanity. Lloyd had destroyed the data fresh and early at 8AM, and worked at rekeying the data the rest of the day. He didn’t leave his desk once.

“So, of course,” Elaine told George, “we waited until the end of the day. Then I walked over, knocked on his cube wall, and he snapped at me. He was just mean, so I put on my sweetest smile, I leaned in, and I said, ‘Hey, Lloyd, you know we have backups, right?’ And the look on his face- that’s why I’m happy with you.”

“Did you get a picture?” George asked. “Because if I don’t get to see that face too, it’s my turn to be upset with you!”

Want to build your desktop, mobile or web applications with high-performance controls? Download Ultimate Free trial now and see what it can do for you!